development of simulation tools â€“ software

BSc VR Surgical Simulation – Software Slide 1

Development of Simulation Tools –Software

Vincent Luboz

Department of Biosurgery and Surgical Technology

Imperial College London

Contents

• Virtual Reality

• VR Surgical Simulation

• Modelling

• Collision detection and collision response

• Performance metrics

• Examples

• Summary and Conclusions


• Virtual Reality


• Modelling



• Examples


Contents


Classic VR Examples?


• Flight Simulators

(here from Airbus company)

Classic VR Examples


• Video Games

(here from Medal of Honor 3 and 4)

Classic VR Examples


What is Virtual Reality?

“Virtual reality is a way for humans to visualize, manipulate and interact with computers and complex data.”

The user interacts with the virtual world and can directly manipulate objects in it.

The technology attempts to immerse the user into a computer generated world representing the reality.


What is Immersion?

• Immersion in an environment (real or virtual) is related to the quantity and quality of sensory data from that environment.

• Immersion in a virtual environment can be gauged by the extent to which the computer shuts out sensations from the real world and accommodates different sensory modalities (vision, audio, haptic).


Examples

Desktop VR SystemWindow on world

Immersa Desk

Immersive systems (HMD) Immersive systems (CAVE)

Telepresence

Mixed reality


Architecture of a VR system

Graphical Display(easy)

Tactile Feedback(very difficult)

MechanicalInterface

(moderately easy)

Rendering Engine(easy)

Computer Models

Geometric (anatomical)(easy)

Physical (biomechanical)(very difficult)

User

Haptics

Audio interfaces can also be included


• Virtual Reality


• Modelling

• Haptics


• Examples


• Hands-on session

Contents


Elements of a VR Surgical Simulation?

Graphical Display(easy)

Tactile Feedback(very difficult)

MechanicalInterface

(moderately easy)

Rendering Engine(easy)

Computer Models

Geometric (anatomical)(easy)

Physical (biomechanical)(very difficult)

User

Haptics

Audio interfaces can also be included


Elements of a VR Surgical Simulation

• Computer Model (geometrical or physical)

Liver

VesselsHips




• Rendering Engine (interactions, collision detection and collision response)

Biopsy Catheterization

BSc VR Surgical Simulation Slide 15




• Interface (haptics)

VSP for catheterization

Phantom for biopsy




• Interface (haptics)

• Performance metrics (measuring the outcome of a virtual surgery and the performance of the trainee)


Simulating Minimally Invasive Procedures

• Involves interaction of several systems

Hardest to achieve accuracy.

• A topic of great interest due to the potential commercial value.

• Example: Gall Bladder (producing bile) and Surrounding organs procedures (Laparoscopic Cholecystectomy)


Simbionix Lap Mentor

Lapsim Laparoscopic Simulator

Sample Commercial Simulators


Immersion Medical Sigmoidoscopy SimulatorBSc VR Surgical Simulation – Software Slide 19

Patient Specific Simulation

Original CTABSc VR Surgical Simulation – Software Slide 20



Segmentation of the vessels



Brach

R. SC

R. CC L. CC

L. SC

AA

RARA

CA

SMA

IA IA

EIAEIA

Fem Fem

•Arch of the aorta (AA),•Brachiocephalic (brach) artery,•Right and left common carotid (RCC and LCC),•Right and left subclavian artery (RSC and LCC),•Renal arteries (RA),•Celiac artery (CA),•Superior mesenteric artery (SMA),•Internal (IA) and external (EA) iliac arteries,•And femoral artery (fem).



Non-pathological Aorta Aortic Aneurysm Aortic Dissection



Model of liver vasculature from fluoroscopy


Simulation Complexity

No deformation - easy

- Bronchoscopy

- Colonoscopy

- Upper GI endoscopy

- ERCP

- Laryngoscopy

- Laparascopic Cholesectomy

Large and Complex Deformations

-

+


• Virtual Reality


• Modelling



• Examples


Contents


Geometric Modelling

• Geometric modelling can be used to reproduce anatomy

• Geometric models can be derived from scan data (CT, MR, US…) and be patient specific

• Applications: walk through the peritoneum, lungs, colon…


Extracting Geometric Models

Segmentation

Mesh Generation

ReconstructionCT Data Set


Creating Walk through

Mesh Model

Render

Texture


Walk Through

Virtual Colonoscopy – University of Hamburg


Spherical model of the Colon

• Easy to create effects such as - spasm,

- inflation of the colon

• Anatomy can be constrained to be representative of a real colon.


Limitation of Geometric Model

• Triangulated models are easy to obtain, but are difficult to deform. (large number of triangles)

• They are suitable for rigid structures (bones, larynx, lung, trachea etc)

• They are less efficient for deformable organs (colon, gall bladder, skin…) and instruments


Modelling Problem - Colonoscopy

TurnView Direction

Turn

View Direction

Turn Rigid Collar

View Direct ion


In a real colonoscopy we might find

If we twist does the loop tighten or the image rotate?


There are many possibilities


Biomechanical Modelling

• One solution to the previous problem is bio-mechanical modelling - however this is fraught with problems.

- What happens when you push the colon wall?

- What dynamic properties does the endoscope have?

- What frictional forces are acting?


Behavioural Models

• For many endoscopic procedures we cannot see the deformation that is caused.

• Thus simulators can get away by modelling only the behaviour of the instrument.


Can we simulate real deformations?

Answer – Almost there but not quite yet!

Certainly not in real time with a high degree of accuracy


Modelling deformations

• The most accurate modelling tool uses the finite element method.

• In the finite element method a surface or volume is divided into small parts called elements.

Element

Face

Node


Typical 3D Elements


Finite Element Model of the Liver

• Geometry extracted from scan data.• Boundary conditions defined initially.• Material properties specified

physically.


Problems

• Real time finite element solutions are possible for small numbers of isotropic linear elastic elements.

• Muscle tissue is not isotropic or linear elastic, and large numbers of elements are required.


Possible solutions

• Precomputation and encoding of a large number of accurate solutions using statistical shape modelling.

• Restriction of the number of degrees of freedom (only appropriate for certain techniques).

• Use of hierarchical approach to solve the local deformation and propagate accordingly.


Basic Interactions

Pressing and dragging

Grabbing and pulling


• Virtual Reality


• Modelling



• Examples


Contents

Collision Detection/Response?


Collision Detection/Response?



Object Database

Geometry

Material

Position Orientation

Force Torque

Contact Information

Collision Detection

Collision Response

Collision Detection/Response


• The algorithms combine the information to obtain the positions in Cartesian inside the virtual environment.

• The algorithm uses position information to find collisions.

• The algorithm reports the resulting degree of penetration or indentation.

Collision Detection


• The algorithm computes interaction forces between objects involved in a collision.

• The algorithm sends interaction forces to the control algorithms.

Collision Response


• Virtual Reality


• Modelling



• Examples


Contents

• Evaluate the performance of the user.

• Give feedback on the task.

• Point to areas that the user needs to improve.

• Derived from a task analysis.

Performance metrics


• Common metrics:- Outcome of the procedure.

- Time spent on the procedure.

- Accuracy of the task.

- Respect of the scenario.

- Comparison with an ideal procedure.

- …

Performance metrics



• Virtual Reality


• Modelling



• Examples


Contents


VR Simulator for Liver Biopsy

Raw image dataLabelled Segmentation

Surface meshTetrahedral mesh

DICOM DATA

SIMULATOR

InteractiveSegmentation Framework

Mesh GenerationPipeline

Deformable Modelling



VR Simulator for Endovascular operation


• Virtual Reality


• Modelling



• Examples


Contents


Summary

We have seen:

• The principles of VR surgical simulator software design

• The concepts of modelling with regard to surgical simulation programs

• How the mechanism of collision is handled by VR simulators

• The basic knowledge and examples of software platforms used for VR simulator


• VR is a powerful technique for visualisation, interaction and planning.

• Surgical simulation systems for training and assessment are increasingly relevant.

• Still improving:- In accuracy (e.g.: real organs elasticity)- In computing time (e.g.: using computer graphic card)- In realism (e.g.: quality of the texture and the rendering)- …

• The haptic rendering allows adding sense of touch to the visualisation immersion

Conclusion


Development of Simulation Tools –Software

Vincent Luboz

Department of Biosurgery and Surgical Technology

Imperial College London

development of simulation tools â€“ software

Documents