DEVAPRIYAN JOHNSON

SUPERVISOR: JAMES HOUSDEN

Project Report

Submitted in partial fulfilment of the Intercalated Bachelor of Science

degree in Imaging Sciences April 2015

AUTOMATED VIEW SELECTION USING A ROBOTIC

TRANS-OESOPHAGEAL ECHOCARDIOGRAM (TOE)

SYSTEM FOR CARDIAC INTERVENTIONS

1

1 Contribution

I worked on this project over a course of seven months from September 2014 to April

2015. I contributed towards the development of a view planning software that has a

high likelihood of eventual future clinical use. My contributions include manual

segmentation of the oesophagus, manual definition of the standard TOE view planes

and evaluation of the accuracy of registration and view plane positioning. In addition

to developing the view planning software, I carried out probe force measurements to

test the maximum force applied by the tip of the probe from manual control of the

TOE handle.

I would like to show my appreciation and give my thanks to my supervisor Dr.

Richard James Housden for the prompt help and guidance. Dr. Housden provided

me with the software to develop the view-planning platform. I would also like to thank

Mr. Shuangyi Wang for providing the remote control to carry out force sensor

measurements.

2

Table of Contents

1 CONTRIBUTION ................................................................................................. 1

2 ABSTRACT ........................................................................................................ 3

3 INTRODUCTION ................................................................................................. 4

TRANS-OESOPHAGEAL ECHOCARDIOGRAPHY ........................................................ 4

CLINICAL USES ................................................................................................... 7

SCANNING PROTOCOL USING STANDARD VIEWS .................................................. 10

PATIENT SAFETY .............................................................................................. 13

REMOTE CONTROL ........................................................................................... 14

ANATOMY OF THE OESOPHAGUS ........................................................................ 17

OVERALL OBJECTIVES OF THE PROJECT ............................................................. 20

4 METHOD .......................................................................................................... 21

VIEW PLANNING SOFTWARE AND VISUALISATION PLATFORMS ................................ 21

4.1.1 3D visualisation in the Unity platform .................................................... 21

SETTING UP STANDARD VIEW WITH REFERENCE HEART MODEL ............................ 23

SEGMENTATION OF THE HEART AND OESOPHAGUS .............................................. 26

4.3.1 Automatic segmentation of the heart .................................................... 26

4.3.2 Tracking the course of the oesophagus ................................................ 28

REGISTRATION ................................................................................................. 29

EVALUATION OF THE VIEW-PLANNING SOFTWARE ................................................ 31

4.5.1 Imaging Datasets .................................................................................. 31

4.5.2 Evaluation ............................................................................................. 31

PROBE FORCE EXPERIMENT .............................................................................. 33

5 RESULTS ......................................................................................................... 36

ACCURACY OF THE REGISTRATION AND THE PROBE POSITION ............................... 36

FORCE SENSOR MEASUREMENT......................................................................... 45

6 DISCUSSION .................................................................................................... 46

ADVANTAGES TO CLINICAL WORKFLOW ............................................................... 46

VIEW-PLANNING SOFTWARE .............................................................................. 47

LIMITING THE CURRENT IN THE REMOTE CONTROL ............................................... 52

7 CONCLUSION .................................................................................................. 55

FUTURE WORK ................................................................................................. 55

8 REFERENCES.................................................................................................. 57

3

2 Abstract

Ultrasonography provides a non-ionising real-time imaging modality to visualise

cardiac interventional procedures and to evaluate the outcome afterwards. In

comparison to Trans-thoracic echocardiography (TTE), the advantage of using

Trans-oesophageal echocardiography (TOE) is that the ultrasound beam does not

have to go through several thoracic structures. The X-ray modality is another gold

standard for cardiac interventional procedures. In several procedures, a combined

approach with TOE is required. The operator of the TOE probe handle is then at risk

of receiving a dose of ionising radiation. Therefore, it would be beneficial to use a

remote control that can control the probe handle and manipulate the knobs to adjust

the position of the probe tip in all the available degrees of freedom. Using remote

control motors rather than manual handling to manipulate the probe increases the

risk of damage to the oesophagus. A view planning software has been developed to

adjust position in the oesophagus and the degree of adjustment of the knobs for

automatic, robust and quick access to precise TOE view planes. This project focuses

on experiments to test the accuracy of the registration of standard views in a

reference model of the heart adjusted to a patient specific model, and the accuracy

of imaging these planes from a probe constrained in the oesophagus. Thirty-one

percent of the available planes accurately provide the required features in the new

model of the heart. The remaining views are only slightly inaccurate and can be

corrected with appropriate adjustment of probe position. Further experiments use a

force sensor to measure the force applied by the probe tip on the oesophagus.

These show that the rear end of the probe exerts six-fold more force than the distal

end. To conclude, results from testing the view planning software developed in this

project are encouraging for eventual clinical use. In addition, the maximum force

exerted from manual manipulation of the knob is under low risk of damaging the

oesophagus: 5N.

4

3 Introduction

Trans-oesophageal echocardiography

The trans-oesophageal echocardiogram (TOE) is a diagnostic ultrasound based

modality used to access anatomy and function of the heart [1]. TOE is commonly

used during cardiac surgeries depending on the complexity of the procedure and

pathology of the patient. The TOE instrument was first introduced into the operating

theatre usage of TOE has increased in widespread

areas such as during cardiac transplantations and major vascular surgeries, as well

as in intensive care and emergency medicine. Intraoperative echocardiographer

receives evolving training to become competent to this form of cardiac imaging.

Echocardiographer for TOE are often an anaesthesiologist, they provide accurate

and timely information to the surgeon, and provide peri-operative management for

patients [2].

TOE is a long flexible gastric probe tube that has a miniature ultrasound transducer

at the tip (Figure 1). The probe that was used in this project is the Philips X7-2t.

Currently in clinical practice, a cardiologist controls the position and orientation of the

TOE probe via the handle. The gastroscope tube also has markings to indicate the

depth of the inserted probe. There are two knobs on the probe handle one knob is

larger than the other one. The larger knob controls the anteflexion and retroflection of

the probe. On the other hand, the smaller knob steers to the left or to the right.

Moreover, the ultrasound beam can also be steered electronically.

5

Figure 1: Labelled image of the structure of the Philips X7-2t (left hand side). Image

on the right hand side demonstrates the probe handle operated by the TOE

echocardiographer [29].

Five degrees of freedom (DOFs) are available by manual control of the probe (Figure

2). Two of these DOFs include rotation of the probe handle and longitudinal control of

the probe. This controls the rotation of the probe head and the depth of probe

insertion into the oesophagus, i.e. rotating the handle rotates the entire probe

including the tip of the probe. The second DOF is the linear transaction of the probe

either forwards or backwards. Gripping the shaft handle, pushing or pulling to

advance or withdraw the probe from the oesophagus respectively [3].The two knobs

on the probe handle control the other two DOFs, which are the steering of the probe

tip. Finally, the fifth DOF is the omniplane rotation (Figure 3). This DOF is controlled

by the operating system rather than the physical displacement of the probe tip.

6

Figure 2: a-d shows the different degrees of freedom of the probe tip. (a) the depth in

the oesophagus, (b) the rotation about the shaft, (c) left-right steering of the tip and

(d) anteflexion -retroflexion of the tip.

Figure 3: Transoesophageal echocardiography (TOE) probes. A, a standard

multiplane TOE probe uses a linear phased array to rotate a 2D plane through 180.

B, a matrix array TOE probe contains piezoelectric elements to scan a 3D pyramidal

volume. The transducer used in this project is B, the matric array TOE probe [3].

7

The probe is popular for cardiac interventional diagnosis and surgical procedures.

The probe is inserted into the oesophagus, which runs immediately posterior to the

heart. Therefore, the ultrasound beam does not have to travel a long distance (only a

few millimetres) to reflect back from the heart to the transducer. This reduction in

distance between the transducer and the heart results in better spatial resolution for

medical diagnosis [4]. The ultrasound waves in TTE have to get through many

structures such as the skin, ribs, fat and lungs. All these structures make the received

ultrasound waves weaker [5]. Therefore, in comparison with other similar minimally

invasive imaging of the heart, such as the traditional TTE, a higher quality image is a

significant advantage to TOE.

TOE used for measuring the size of internal organs, chambers and vessels. However

since the majority of the patients are expected to have undergone TTE before TOE,

measurements from TTE are used in these cases. This is because the evidence base

is higher in TTE in comparison with TOE [5]. In addition, some measurements are

difficult to achieve in TOE. This is because of the proximity of the transducer, e.g. LA

dimensions. In addition, some measurements are also more prone to error in case of

images obtained off axis, e.g. LV dimensions. Despite the inappropriateness for

measurements, some measurement are usually more precise in TOE, such as the

aortic root size and the annular dimensions.

Clinical uses

Along with the diagnostic uses of TOE, including detection of prosthetic valve

dysfunction and endocarditis, during surgical procedures TOE guides the operation

by providing US imaging of the heart and the structures of the heart, such as the

leaflets of the heart valves and the subvalvular apparatus [6]. TOE is commonly used

8

to evaluate the life-threatening hemodynamic disturbances that may occur during a

surgical or non-surgical procedure. Intraoperative use of TOE included monitoring

the valves and congenital heart disease repairs.

TOE use also includes monitoring left ventricular function during coronary artery

bypass graft (CABG). In addition to confirming a suspected diagnosis, the benefit of

TOE scanning during the procedure involves assistance in positioning of

intravascular devices, early identification of ischemia and altering surgical

management or medical therapy. In patients from Category 1 (28%) (Table 1) use of

TOE is more beneficial and therefore results in alteration of the therapy in

comparison with Category 2 (14%).

One of the strongest clinical benefits is that Intraoperative TOE aids in the evaluation

of the mitral valve during repair. In an assessment of mitral valve morphology using a

systemic approach, there is good agreement between the surgical finding and TOE

(92% agreement in comparison with patients not systemically studied) [6]. Routinely

using TOE during valve repairs and replacements is certainly an advantage in

adjusting the management and providing an effective postoperative care.

9

Table 1: Lists the peri-operative uses of TOE [29].

Patients receiving left ventricular assist device (LVAD) implantation treatment

undergo TOE as it plays a vital role in the selection and surgical management. An

intact aortic valve is essential for proper LVAD function. For this purpose, TOE is

considered a beneficial tool to assess the function of the aortic valve. This is

because an incompetent aortic valve would result in significant retrograde flow.

Shunting of right-to left is produced across an atrial septal defect (ASD) and Patent

Foramen Ovale (PFO) because of activation of the LVAD implantation. For this

10

reason, it is important to correct the intra-cardiac shunts, PFO and aortic

regurgitation before implanting and activating the LVAD. The mechanism of LVAD is

detected by TOE. This includes obstruction by the cannula or valve regurgitation

through the inlet or the outlet. TOE also aids in the optimisation of the device

performance and guides the operator if the LVAD is deterring [12].

The success and the safety of many catheter based approaches for treating

congenital heart malfunctions is improved by using TOE images. Catheter based

approaches are used to treat 40% of congenital heart malfunctions [13, 14]. It is

appropriate to use TOE for determining the relation of the veins and a septal defect

to the adjacent valves. TOE is used to assess the morphology of the valves and

hence it is an appropriate test to select patients with PFO closure and percutaneous

ASD [15]. Guiding the device during the positioning and placing is aided by the

assistance of TOE during such procedures. After resolving the congenital heart

malfunctions, TOE is used to detect and assess the severity of residual effects, such

as interference with valve function and obstruction of pulmonary venous pathway.

Scanning protocol using standard views

Recent imaging technologies have provided the opportunity to obtain real-time (RT)

3-dimensional (3D) echocardiography of the heart using the transoesophageal or

transthoracic (TTE) matrix array probe that provides online 3D images [3-5]. There is

analytical software providing quick offline reconstruction of these 3D dataset. This is

beneficial in further assessing structures such as the mitral valve and quantifying the

function, e.g. left ventricle.

3D TOE depends on volume datasets in comparison to 2D TOE that depends on

standard imaging planes [7]. The TOE trained echocardiographer must manipulate

11

the view to orient the 2D or 3D images to ensure to obtain the basic standard views.

This indicates that that heart can be viewed in several normal and pathological

perspectives [8]. This is an essential visual aid to understand the anatomy of a heart.

In order to minimise the dataset and to provide a framework for performing TOE

imaging in different clinical settings, the British Society of Echocardiography has

come up with 20 standard views of TOE. This also recommends a sequence to

perform a complete TOE imaging to specify areas of interest in the heart (Figure 4)

[2]. Despite providing a systematic approach to acquired TOE imaging there are

specific issues at each views. In addition, it has been recognised that not all views

can be imaged in all patients, especially some poorly tolerated views in which the

probe can cause discomfort to some patients, even in the post recovery phase from

general anaesthesia (GA), e.g. deep gastric.

Therefore, the operator acquiring the TOE images can decide to omit a view taking

into account the balance between risk of insufficient data versus patient comfort and

safety. In recent days, cardiologists, cardiac physiologists or cardiothoracic

anaesthetists operate the TOE probe to obtain the required views for cardiac

interventional procedures. In addition to that, there is medico legal justification to

ensure the format of the standard views has been followed precisely in research

studies.

12

Figure 4: Shows the 20 standard views shortlisted to examine all aspects of TOE

comprehensively. The icon adjacent to each view indicates the approximate

omniplane angle indicators. ME Mid-oesophageal, LAX longitudinal axis, TG -

transgastric, SAX short axis, AV aortic valve, RV right ventricle, asc -

ascending, desc descending and UE upper oesophageal [2].

13

This approach has been appropriate to avoid missing important diagnostic

information that may not appear in preoperative TTE. This means the cardiac

interventional TOE has to be well coordinated in order to complete the entire format

of the study. For interventional procedures that require the chest to be open it is

recommended to obtain most of the data before opening the chest as the images

may be affected afterwards, e.g. dimensions of the tricuspid annulus.

In addition, all TOE procedures are carried out on the patient under GA. Therefore,

the clinician must take into account that patients physiology might vary due to general

anaesthesia, vasoactive drugs and fluid status. It is important to consider these

principles in deciding to acquire TOE images before listing the patient for surgery. For

example, the severity of the mitral regurgitation varies according to the physiology at

the time of the study.

Patient safety

The duration of the TOE is usually 45-60 minutes, which includes preparing the

patient, e.g. both oral and written consent, cannulations, and could involve a

preliminary TTE [9]. Some clinical circumstances may require more focus on image

acquisition and these procedures may take longer based on clinical judgement.

The pressure exerted by the probe can damage the oesophagus. In comparison with

other cardiac interventional modalities, TOE is relatively safe [9, 11]. However,

inserting and manipulating the probe inside the oesophagus can potentially result in

oropharyngeal, oesophageal, or gastric trauma.

This indicates that TOE is a semi-invasive procedure with potential for serious

complications [10]. Therefore, since patient safety is the crucial first priority, in order to

14

ensure these complications are avoided there are mandatory routine checks, e.g.

oesophageal stricture, loose teeth, previous gastro-oesophageal surgery, etc.

plan is required. In this way, identifying the risk factors and manoeuvring gently and

cautiously prevents severe and life threatening compilations.

There are also safety aspects from general anaesthesia that might affect the TOE

acquiring procedure. Firstly, only a trained anaesthetist provides sedation for the

patients. Monitoring the blood oxygen saturation before and after the procedure and

continuously checking the saturation levels during the entire length of the TOE

procedure is also mandatory [11]. The equipment to resuscitate must be fully available

if saturation levels seem to decrease. The echocardiography lab must also provide

protocols on decontamination of the probe after each procedure and sterilisation of

all the equipment that goes along with the TOE probe. These documents must also

agree with protocols from the local infection control department. This ensures the

safety of the patient and follows the code of ethics that the treatment received is in

the best interest of the patient.

Remote control

Interventional procedures usually include a combination of x-ray fluoroscopy. Certain

interventional procedures require patients undergo x-ray tests alongside acquiring

TOE ultrasound images. This often involves a skilled operator in the path of the x-ray

to receive a dose of ionising radiation [10]. This is undesirable for long cardiac

interventional procedures. This implies that the longer the procedures the more

significant dosage is received by the TOE operator. Wearing heavy lead clothing is

also a burden for the operator to protect against x-rays.

15

Another disadvantage of manual operating of the TOE probe handle is the demand

of highly trained echocardiographers. This issue inflict financial burden upon the

firms to organise regular training for the staff operating the TOE device. These are

the disadvantages of using TOE imaging for cardiac interventional procedures.

A remote control to operate the TOE probe would be an ideal solution. This is

because the remote control can be operated from a relative distance or even behind

lead protection. The remote control operated robotic system is an advantage as it

allows longer use of the probe (as long as the interventional procedure takes)

without a significant radiation exposure to the operator.

In addition, using a view-planning program that automatically navigates to the

requested plane is a possible solution to address the demand for highly trained

echocardiographers. A semi-automatic control of the probe reduces the need for

skilled radiographers. This as this eliminates the cost

of arranging termly training for the operator to remain up-to-date with the imaging

techniques. These are potential solutions for addressing both these major problems

associated with TOE imaging.

This project focuses on developing an automatic view planning approach in which

the probe is automatically positioned to the appropriate locations in response to a

requested specific view of the heart. The results from the view planning experiments

will form the basis of using 3D ultrasound data to register with the robotic coordinate

system. Furthermore, incorporating these functions to go with the pre-planned views

the remote control software will be developed.

16

Figure 5: Shows the TOE probe handle mounted to the remote control. This image

also explicitly show the railing to bring about the longitudinal protraction and

retraction of the probe tip as the motors slide on the railing.

Existing robotic system - The remote control robot manipulates the probe handle with

four out of the five degrees of freedom (DOFs) available using the manual

manipulation. These include, the rotation of the two knobs controlling the anteflexion

retroflexion and left-right steering of the probe tip. The protraction retraction and

rotation of the probe is controlled by the railing in the built in remote control (Figure

5). The software that runs on the PC controls the probe. The software synchronized

to a joystick or a gamepad connected with the computer allows an opportunity for

wireless control of the remote control. The remote control has been tested to find out

the correct working of the control and the mechanisms. A gamepad or joystick is an

17

alternative ideal choice for controlling the robotic remote control. The robot can be

manually controlled remotely, either on a PC or via a joystick (Figure 6, 7).

Figure 6: Shows flowchart of the overall interfacing method.

Figure 7: Diagram of the gamepad input for TOE robot control [29].

Anatomy of the oesophagus

The heart and the lower part of the mediastinal oesophagus is in close anatomical

relation to each other. The oesophagus is posterior to the heart and is separated

from the left atrium by the pericardium [16, 17]. In this way, this aspect is very useful in

obtaining an echocardiogram by inserting a transoesophageal probe.

The oesophagus is a tube-like fibromuscular organ through which food passes to get

from the pharynx to the stomach, with the assistance of peristaltic contractions. The

Gamepad/Joystick Software in PCRemote control

motor

18

lumen of the oesophagus is quite flexible such that it changes shape to give way for

the food that is passing though [18]. From the lumen to the outside, the oesophagus

consists of mucosa, sub-mucosa, and then the layers of muscle fibres within the

layers of fibrous tissue then finally connective tissue at the outermost layer.

Oesophagus is a thoracic organ, whereas the stomach is an abdominal organ. The

structure of mucosa in the oesophagus is different from that of the stomach.

Oesophageal mucosa consist of stratified squamous epithelium. On the other hand,

the stomach mucosa consist of simple columnar epithelium. This histological

characteristic of the organ is an important consideration to understand the safety of

the patient during the interventional procedure.

Oesophagus has two subtle curves at the commencement at the superior thoracic

aperture and near the descending thoracic aorta. Oesophagus has three

constrictions along its course at the level of cricophargeoal sphincter, between the

aortic arch and left main bronchus and finally as it pieces through the diaphragm [19].

Other than the two area of attachment at the top at the cricophargeoal level and at

the diaphragm, the organ is mobile [21]. This applies even in trunk changes in the

patients from forward to backward, lateral shifts and supine to prone positions [22].

These are the main reason for considering oesophagus rather than any other organ

in the mediastinum for TOE imaging.

However, the relations and common path for pain fibres from both organs via the

sympathetic trunk leads to difficulty in determining the origin of pain. This becomes

an issue as the safety aspects of inserting the semi-invasive TOE probe and the

remote control of the inserted probe has a potential to damage the oesophagus and

the surrounding mediastinal structures [20].

19

Oesophagus takes a variable course within each individual. Most commonly

oesophagus is observed to be located rightward of the spine, compressed between

left atrium and the spine. A groove behind the atrium bound by the spine and the

aorta is commonly observed. Oesophagus is also compressed between the left

atrium and aorta. This compression is usually observed leftward of the aorta.

Furthermore, the position of the oesophagus to the LA is most commonly adjacent to

right PV antra and ostia, between the right and left PV, or the left PV antra or ostia

[23, 24, 25, 26]. This evidence shows that the course of the oesophagus has subtle

variation between individuals

In addition to the variation in the course, the oesophagus is also very mobile within

the individual. Relative to the position of the LA the position of the oesophagus can

be dynamic. It shifts laterally several centimetres in relationship to the LA because of

changes in the patient position from right to the left patient recumbent position [27] or

deglutition and peristalsis in the awake or sedated patient [28]. However, the

oesophagus position is constant in a resting position, i.e. in a prone patient lying

down in a bed the position of the oesophagus remains consistent.

The mobility of the oesophagus can affect the position of the probe, and hence the

view planes, compared to the position assumed by the model. In this way, the

physiological or pathological variation in the oesophagus and lateral shifting of the

highly mobile oesophagus has implication for the view planning software developed

in this project. Along with the view planning aspects of the project, the TOE safety

profile was reviewed by experiments to determine the maximum force exerted by the

tip of the probe.

20

Overall objectives of the project

I. Develop a clinically practical semi-automatic view-planning platform.

II. Determine the safety of the robotic system by measuring the forces applied.

21

4 Method

View planning software and visualisation platforms

The plan for visualising the standard views using the software is to segment a heart

model from pre-acquired MRI data, and manually segment the course of the

oesophagus. These two organs will be combined in a view platform software in

which the standard view slices can be selected and visualised. Then compare the

standard views in the Reference heart model with Experimental heart MRI models.

This aspect is beneficial as it provides an opportunity to develop algorithms that

determine probe position automatically by selecting a view.

4.1.1 3D visualisation in the Unity platform

The automatically segmented heart and the manually segmented oesophagus are

visualised using a gaming platform, known as Unity, developed by Unity

technologies. Unity is a 3D visualisation software that allows us to develop 3D video

games for websites, desktop platforms, mobile devices and consoles (Figure 8).

These features not only enabled us to visualise the segmented heart and the

oesophagus but also to slice the segmented heart to view the standard views.

22

Figure 8: shows a screenshot of the unity software visualising a 3D heart model.

The heart data is loaded onto the Unity platform using the five-step process. The first

step involves loading the heart surface file for the dataset under trial. We have two

datasets to trial in the experiments. The next step is to load the corresponding

oesophagus using the similar file type. After the heart and the oesophagus

segmentation has been loaded, the probe direction has to be adjusted. This is

because the software does not identify whether the probe is facing the right direction

along the oesophageal course. Therefore, the direction of the probe is adjusted

(Figure 9). This step is to ensure the

digital segmented probe is aligned in the direction that simulates the direction of the

actual TOE probe, i.e. from superior to inferior.

23

Figure 9: Shows the screen shot of the Unity platform after loading the model of the

heart, the oesophagus and reversing the probe appropriately to align the direction of

the probe tip. The panel in the left hand side also shows the probe control

parameters that simulate the actual control of the TOE probe tip

Setting up standard view with Reference Heart model

The Department of Anaesthesia at the Toronto General Hospital has developed a

Virtual TOE interactive educational tool. It is an online website designed to facilitate

learning of TOE. This educational tool provide details on brief descriptions of each

view including bullet point listing of the features and structures expected from each

view (Figure 10). The website also states detailed information to obtain each

standard views. This information was used as a guide to set up the standard views.

24

Figure 10: Shows the perioperative interactive educational website that provides 3D

visualisation on the standard views. The website provides a brief description of the

expected structures on each standard view and the steps to obtain the views.

URL - http://pie.med.utoronto.ca/TEE/

A line drawn along the course of the oesophagus in the Rview software. The

landmarks are manually located in the software and the traced oesophagus is placed

next to the automatically segmented heart model of the same MRI data. Then find

the standard view slices using the Unity 3D viewer. In this way, the standard views

are set up using a heart model from the first MRI data. The functions in the software

to control the probe along the course of the oesophagus are very similar to the DOFs

available for the actual probe. This is to simulate obtaining the TOE image using

http://pie.med.utoronto.ca/TEE/

25

manual manipulation of the probe. These functions include depth as a percentage of

the segmented length of the oesophagus, axial rotation, left/right steering,

anteflex/retroflex steering and the image plane steering in degrees.

Using the software developed in Unity, the standard views are selected from the

Reference heart model of the automatically segmented heart. In order to set up

standard views, the Reference model of the automatically segmented MRI heart data

and manually segmented oesophagus is loaded. This is to take into account the

position of the TOE probe to image the specific view. For example, the mid-

oesophageal four-chamber view would involve the probe to be in the middle of the

oesophagus, whereas the transgastric two-chamber view would involve the probe to

enter the fundus of the stomach. In this way, adjusting the probe tip position has

enabled to select the thirteen standard views in the heart model.

Only thirteen standard views were available for the Reference heart model. This is

because the other seven view planes require additional features extending further

from the heart. The aorta is the core requirement for imaging six out of these seven

planes. View planes that include the ascending aorta (AA) are the upper-

oesophageal AA LAX, SAX, mid-oesophageal AA LAX and SAX. Moreover, the view

planes involving the descending aorta are the mid-oesophageal descending aortic

SAX and LAX. The automatic segmentation of the heart limited the full extension of

the aorta. Finally, shortened length of the oesophagus post segmentation limited the

unique deep gastric LAX plane. The oesophageal segmentation could not be

extended all the way to the stomach and the fundus of the stomach. Therefore, the

deep gastric LAX view plane was omitted as well. Despite these limitations, thirteen

views were reasonably suitable views for the Reference heart model.

26

Segmentation of the heart and oesophagus

4.3.1 Automatic segmentation of the heart

The algorithm to automatically segment the heart to make the 3D visualisation was

first described by Peters et al [30]. 3D MR images are segmented and converted into

a 3D visualization of the heart. The main advantage of using automatic segmentation

is faster and easier than manual segmentation. Manual segmentation of the heart

could take several hours. It is clinically not feasible, as it consumes considerable

amount of staff time. However, automatic segmentations have several

disadvantages that include increased image noise, patient variability, low contrast

between the surrounding tissues and myocardium, spatial magnetic field

inhomogeneities and lack of grey level calibration.

There are several techniques potentially applied to segment the cardiac MR images.

Including active shape models, deformable models, active appearance models, level

sets, active contours and atlas-based methods. Out of all these techniques, shape

constraint deformable models formed the basis of the automatic segmentation

algorithm. The algorithm not only describe the automatic segmentation of the

ventricles but also the other two upper chambers of the heart, ventricular

myocardium, the pulmonary vessels and trunks of the aorta. This developed

algorithm is applied on MR images acquired in a stack of slices in the long axis

and/or short axis view. The dataset was nearly isotropic voxel resolution with static

cardiac (3D) image volume acquired with steady state free precession MRI. One of

the advantages of this automatic segmentation algorithm is that the position of the

heart is unknown prior to the processing. This is a beneficial aspect as the algorithm

27

can work out the borders by itself. In addition to that, the surrounding tissues are also

visible since the field of view is not restricted to just the heart alone.

Figure 11: Shows two different views of the heart model using the standard 3D heart

MRI after applying the automatic segmentation algorithms [30].

The anatomy of the automatically segmented heart extracted from the MR image

volume includes the four-chambers, the trunks of the aorta, the myocardium, the

pulmonary artery and the pulmonary veins (Figure 11). The segmented structures

are combined together using deformable mesh features to make the heart model.

The mesh is composed of a combination of numerous triangles combined to make

vertices. This involves a complex function to combine surfaces with three of more

junctions [21]. Prior knowledge of the shape is useful in estimating and closing up the

missing boundaries. In this way, using the shape-constraint deformable adaptation

the segmentations are stabilised and made into a regular shape [22].

28

4.3.2 Tracking the course of the oesophagus

The oesophagus was segmented using the viewing software Rview. Rview is a

software that integrates a number of 3D/4D data displays. Using the normalised

mutual information, it comprises fusion routines integrated with 3D rigid volume

registration. It also includes several features to carryout interactive volume

segmentation. Rview also includes painting functions to analyse the structural data.

Out of these features, the most relevant function to this project is placing landmarks

in the 3D MRI data that is used for the segmentation of the oesophagus (Figure 12).

Figure 12: Shows the screenshot of the Heart MRI and the loaded completed

landmarks of the course of oesophagus in the Rview visualisation software.

29

The oesophageal course implies the course of the inserted TOE probe. Firstly, load

the heart MRI data on the Rview platform. Then using the segmentation features

with the aid of the cursor, landmarks are selected along the course of the

oesophagus. The landmarks connected in one path by manual efforts to click on the

middle of the oesophageal diameter when selecting the landmarks using the cursor.

These are the steps to perform the manual segmentation of the oesophagus for all

three MRI data sets in the Rview platform. The landmarks of the oesophageal course

is saved for each of the heart MRI (Figure 12). This is accessed later on for the 3D

visualisation on the Unity platform.

Tracking the course by clicking along the centre of the oesophagus is an extremely

simple task. A minimum knowledge and understanding to identify the oesophagus in

the mediastinum is more than adequate to perform the task. This aspect is a major

clinical advantage, as the task does not demand a highly qualified clinician or a great

deal of time to carry out the task.

Registration

After selecting the standard views, experiments were performed to compare the

reliability of the view planning software in producing a similar view plane to that of the

required standard view. The view planning software was used to make this

comparison, as it provided all the relevant features and appropriate functions that

enabled the code to set up the programme to be written. The first step of the

comparison involved loading up the model of the heart segmented from the second

MRI data set. The aim of the experimental MRI data is to simulate the clinically

obtained patient heart model. The oesophagus of the experimental MRI dataset and

the standard view of the Reference dataset that was previously saved. In this

30

instance the standard views of the first heart acts as the reference to the standard

views registered onto the Experimental heart models.

Iterative closest point (ICP) registration was used to combine the Reference heart

model with the Experimental heart data. ICP is a point feature based registration in

which the difference between the two clouds of points is minimised using the

algorithm. The registration was affine (12 degrees of freedom), involving translation,

rotation, scale and shear of the model. Affine transformations have the property that

a 3D plane stays as a plane after deformation, which is useful for not distorting the

target image planes.

Root mean squared error (RMSE) is a quantitative measure of displacement between

all the points that occurs even after applying ICP (Figure 13). RMSE is frequently

used to compare the accuracy of registration. RMSE is a reliable measure that can

be used to quantify registration errors between models. The error in this case is the

distance between points on the Reference model to the closest point on the

Experimental heart model.

In this process, the points from the Reference heart model deforms and compute to

match the nearest point in the point set of the Experimental heart MRI data. The

model is divided up into sections (LA, LV, RA, RV, Aorta, Pulmonary artery,

Myocardium) as provided by the automatic segmentation. For increased robustness

in the registration, points in a particular section of the Reference model are matched

only to points in the corresponding section of the new model.

Figure 13: shows the formula to calculate the rooted mean standard error (RMSE).

31

Analytically the transformation based Reference model to the patient-specific Heart

models. In this way, best match to the Experimental heart model obtained by the

transformation of the points in the Reference heart model. The ICP algorithm

includes iterative revisions of the transformation to minimise the distance of the

reference point cloud of the Reference model from the source, first MRI out of the

datasets. The process is repeated until maximum possible convergence.

Evaluation of the view-planning software

4.5.1 Imaging Datasets

The MRI datasets that were used in this project were acquired from pre-procedural

scans. All datasets are standard patient MRIs acquired with a Philips Achieva 1.5T

MRI scanner using an ECG gated sequence and balanced SSFP. The sequence

details of the Reference heart are: TR/TE: 5.22s/2.61ms, with 90 flip angle, slice

thickness of 2.74mm, image size of 256 x 256 x 120 and voxel size of 1.34mm x

1.34mm x 1.37mm. The sequence details of the Experimental heart one are: TR/TE:

4.34ms/ 2.17ms, with 90 flip angle, slice thickness of 3.00mm, image size of 224 x

224 x 119 voxels and voxel size of 1.34mm x 1.34mm x 1.5mm. Finally, the data

sequence details of the Experimental heart two are: TR/TE: 4.63s/2.32ms, with 90

flip angle, slice thickness of 1.56mm, image size of 432 x 432 x 180 voxels and voxel

size of 0.79mm x 0.79mm x 0.78mm.

4.5.2 Evaluation

The view planning experiments are in two stages in viewing the new heart under trial.

The first step evaluates the target plane, i.e. the adjusted standard view in the new

model of the heart. This target view is derived from the standard view setup in the

32

initial MRI dataset (Reference Model) adjusted to the Experimental heart model one

and two using the ICP registration. This permits the experiments to test the view

planes.

First experiment is to check the accuracy of the registration. The first experiment

involves using the registration and visualising the actual view plane (Figure 9). The

data collection step involves visually analysing the degree of matching of the actual

view plane to the requirement of the target plane from standard views. The

information on the requirements of the each standard views is from the electronic

interactive TOE educator tool designed by the Toronto hospital (Figure 10). The

qualitative approach involves categorising each standard view into four groups (table

2). The reason for using a qualitative approach through visual grading is that the

required proximity in mm for an adequate match is unknown. In addition, the required

closeness may vary between different standard views. For this reason, the qualitative

approach gives information that is more useful.

Table 2: Shows the classification criteria based on the degree of match to the

standard view requirements. These guidelines are used when visually observing and

categorising each view plane

Category Degree of match to the standard view requirements

A All required features are present in the image

B Close but can be corrected by adjusting probe parameters. For

example, in the Mid-oesophageal four-chamber view, all four

chambers present but the view missed the apex.

C Same as Category B, but cannot be easily corrected by

adjusting the probe positions.

D View is completely wrong

33

View planes that fall in the Category B are the planes that are easily fixed.

Categorising to group B planes also requires suggestions for suitable adjustments to

make these changes.

The target plane for a standard view is already defined in the new heart by the

registration step. The aim is to position the probe in order to view this plane. The

iterative optimisation adjusts the five parameters, which adjusts the probe position

and hence the view plane position. The aim is to find a set of parameters that position

the actual view plane (which depends on the probe position) as close as possible to

the target plane. The iterative optimisation measures distances between a grid of

points defined in the target plane and the actual plane. The set of parameters that

minimise these distances is taken as giving the best probe position.

In the way, the probe position is taken into account for aligning the image plane in the

second experiment. The second experiment is also graded according to the

categories illustrated in table 2.

The conclusions drawn from both experiments are beneficial as they can be used to

programme the remote control to position the probe tip in the correct place to image a

plane that is as close as possible to the target plane.

Probe force experiment

The robot comprises current sensors that monitor the current applied to control the

steering of the probe tip. This provides feedback for force applied at the tip of the

probe and therefore helps prevent oesophageal damage. As part of the study, the

aim of the experiments carried out is to determine the maximum forced exerted by

the probe.

34

Figure 14: Showing the setup of the initial force sensing experiment using the

silicone tube simulating the features of the oesophagus.

The experiments to determine the force exerted on the oesophagus were performed

using a silicone tube with outside dimension (OD) of 26mm, inside dimension (ID) of

20mm and length of 250mm to mimic the basic characteristics of the oesophagus,

such as a mobile and flexible tube (Figure 14). The force was measured using a six-

axis Nano-17 sensor.

Figure 15: Shows the Nano-17 measuring the force generated by the rear end

(above) and the distal (below) end of the tip of the TOE probe.

35

The probe is interested into the silicone tube (Figure 15) to measure two aspects of

the generated force. The force generated on the Nano-17 resistance sensor by the

distal end of the probe tip was measured first. Then the force generated by the rear

(proximal) end of the probe tip was also measured. Both the distal and proximal end

measurements were carried out using three individuals to acquire data on the

variation in the force exerted by manual manipulation of probe handle and the

maximum force was noted. Each individual applied the maximum force achievable.

The results from both parts of the probe provide insight into adjusting the maximum

current limit to set on the robot motors to manipulate the probe safely.

36

5 Results

Thirteen out of the twenty view planes were appropriately marked in the model of the

Reference heart obtained from the automatic segmentation algorithm. These thirteen

views are listed in table 3.

Mid-oesophageal four-chamber Mid-oesophageal Bicaval

Mid-oesophageal two-chamber Transgastric Mid-SAX

Mid-oesophageal mitral commissural Transgastric two chamber

Mid-oesophageal LAX Transgastric basal SAX

Mid-oesophageal aortic valve SAX Transgastric LAX

Mid-oesophageal aortic valve LAX Transgastric right ventricular inflow

Mid-oesophageal right ventricular inflow-outflow

Table 3: Shows the thirteen view planes that were the heart models.

Accuracy of the registration and the probe position

All the mid-oesophageal view planes were matched well in terms of imaging the

required features with a few exceptions where features were missing or limited from

visualisation by the segmentation and processing of the heart model from the MRI

data. The registration of the Reference heart model to the Experimental heart model

one has a RMSE of 2.414mm2. The registration for the Reference heart model to the

Experimental heart model two has a RMSE of 3.644mm2.

37

Figure 16: Visualization of the Reference heart model (red) registered with the

experimental models (blue) - Heart model one (left) and Heart model two (right).

The registration between the Reference heart model and the experimental models

has been successful. The RMSE is smaller in the registration to Experimental heart

model one in comparison to Heart model two. A MSE score of zero indicates perfect

flawless registration. Moreover, a high MSE value indicates poor accuracy in

registration. This indicates that the registration is more accurate in the Experimental

heart model one in comparison with Heart model two.

Ideally, the red mesh should be perfectly aligned with the blue mesh to show an

accurate registration. Inaccuracies in the registration of the Reference heart model to

the Experimental heart model one are relatively minor (RMSE= 2.414mm2). The

misalignment is most noticeable lateral to the RV and the ascending aorta.

Registration of the Reference model with the experiential Heart model two has a

RMSE score of 3.644mm2. This value is higher and the misalignment is

demonstrated in Figure 16. A significant misalignment is noticeable at the level of