Maria Grazia Pia

Benchmarks of medical dosimetry simulation on the grid

G.A.P. Cirrone1, G. Cuttone1, F. Di Rosa1, F. Foppiano2, A. Lechner3, P. Mendez Lorenzo4, J. Moscicki4, M.G. Pia5, M. Piergentili5, G. Russo1

1INFN Laboratori Nazionali del Sud, Italy2IST, Genova, Italy

3Technical Univ. Vienna, Austria 4CERN, Geneva, Switzerland

5INFN Genova, Italy

CHEP 2007Victoria, BC, Canada3-7 September 2007

Maria Grazia Pia

Monte Carlo methods in oncological radiotherapy

More precise than approximated analytical methods More precise than approximated analytical methods used in commercial treatment planning software systemsused in commercial treatment planning software systems

BUT

Too slow to be realistically usable in the clinical practiceToo slow to be realistically usable in the clinical practice

Variance reduction techniques Variance reduction techniques (event biasing)(event biasing)

Inverse Monte Carlo methodsInverse Monte Carlo methods

Analytical transport methodsAnalytical transport methods

“ “Fast simulation” techniquesFast simulation” techniques

ParallelisationParallelisation

Reduce simulation Reduce simulation execution timeexecution time

The GRID?

Maria Grazia Pia

This projectThis project

Practical approach to the problem

Let’s take a few representative dosimetry simulation use casesLet’s run them on the gridLet’s evaluate (quantitatively) gains and drawbacks in realistic situations

Three Geant4 Advanced examples of dosimetry simulation covering major techniques in oncological radiotherapy

– Brachytherapy– Hadrontherapy– LINAC for IMRT (Intensity Modulated RadioTherapy)

LCG infrastructure, Geant4 Virtual Organization– The exercise will be repeated one step further as Mr. Nobody

No introduction on grid, EGEE and other flavours, LCG, medical-related grid projects, their impact etc.

Maria Grazia Pia

brachytherapybrachytherapy

Leipzig applicator

MicroSelectron-HDR source

3 m m ste e l c a b le

5.0 m m

0.6 m m

3.5 m m

1.1 m m

Ac tive Ir-192 C o re

0 10 20 30 40 500,0

0,2

0,4

0,6

0,8

1,0

1,2 Simulazione Nucletron Misure

Dose %

Distanza lungo Z (mm)Distance along Z (mm)

SimulationNucletronDataExp. data: F. Foppiano et al.,

IST Genova

Relatively “fast” simulation

~ 7 CPU hours on an “average” PC to produce meaningful statistics for

clinical studies

Italian National Institute for

Cancer Research

Geant4 brachytherapy Advanced Example

Maria Grazia Pia

hadrontherapyhadrontherapy

Accurate reproduction of the CATANA hadrontherapy

facility at INFN LNS, Catania, Sicily, Italy

Geant4 hadrontherapy Advanced Example

Geant4 geometry and beam primary generator implementation developed by INFN-LNS CATANA co-authors(Partial) code review by A.L. and MGP

See related talk in Event Processing

Session

The experimental data shown in this plot have been taken at the CATANA hadrontherapy

facility at INFN LNS, Catania by G.A.P. Cirrone, G. Cuttone, F. Di Rosa, G. Russo.

No credit to other co-authors and their institutes for these data.

Production aimed at validating physics modelling of the simulation

~ 150 CPU hours on an “average” PC to produce meaningful physics results

~ 16 hours for calibration results

Maria Grazia Pia

Medical LINAC for IMRTMedical LINAC for IMRT

High demand of CPU resources for meaningful statistics

(e.g. for treatment planning verification)

≈ tens of CPU-days

Geometry developed by M. Piergentili

Italian National Institute for

Cancer Research

Exp. data: F. Foppiano et al., IST Genova

Geant4 medical_linac Advanced Example

Maria Grazia Pia

RequirementsRequirements

Submit jobs to a Grid system

Plug-in Geant4 application

Split into tasks (without biasing the results)

Schedule tasks intelligently

Hide the underlying complexity to the end user

DIANE

Ganga

LCG

Geant4AIDAiAIDA

+ +

++

Maria Grazia Pia

DIANEDIANEGeant4Geant4InterfacInterfacee

Interface class which binds together

Geant4 application and DIANE framework

DIANEMaster-Worker

architecture

Maria Grazia Pia

Computing environmentComputing environmentIdentification of resources

– The site must support the Virtual Organization (Geant4)– Shared file system accessible through VO_GEANT4_SW_DIR

environment variable– Operating system and architecture compatible with Geant4

Software installation– Geant4, CLHEP, AIDA, gcc– Problems encountered: missing write permission, user authentication etc.– Requires familiarity with grid middleware

Requires experienced userOne must do one’s own installation on the grid

Interface to grid middleware (e.g. lxplus) must be set-up, otherwise additional (not trivial) installation needed (e.g. gLite)

Maria Grazia Pia

Problem set-upProblem set-up

Determination of the total # of events to be produced– Defined for each application on the basis of the desired statistical

precision of dosimetric variables relevant to each use case

Splitting into tasks – Task = smaller group of events to be produced

Perform tasks sequentially on a worker node

Use a range of worker nodes in parallel in a grid environment

Critical issue

Maria Grazia Pia

Task splittingTask splitting

Important for applications with short execution time – Time elapsed between submission of grid jobs and

beginning execution on a worker node comparable to total duration

High granularity → more effective use of CPU resources– Drawbacks: in general, larger output size to deal with

Task size(execution time)

# workers involved

balance

Maria Grazia Pia

Splitting optimizationSplitting optimization

Goal: fastest simulation results

Many small tasks – 20000 events, ~ 25 s each

40 worker nodes

Higher # nodes would not be more effective

– Time for last registered workers to become active might be higher than total duration

Goal: sensitive simulation study– beam energy calibration

1 task = 1 worker– Not the fastest option

Compromise with large disk space requirements for output

Simplify distribution of random number seeds passed to tasks

“Fast” use casebrachytherapy

“Intensive” use casehadrontherapy

“One size DOES NOT fit all”

Maria Grazia Pia

Baby-sittingBaby-sittingSelection of resources

– Availability, priority

Set-up a list of Computing Elements– Passed to Ganga through DIANE

Criteria for selection– State of Computing Element (production, draining mode)– Number of CPU allowed to VO at a given site– Estimated waiting time in batch queue– Estimated additional workload on given nodes– CPU speed

Relatively easy retrieval of information– Used for selection criteria

Must be done on a

regular basis

Automated tools not always effective

Maria Grazia Pia

Effects of delayed worker registrationEffects of delayed worker registration

fast worker registration

aslow worker registration

b

Unsatisfactory conditions: delayed worker registration,

worker errors (crash, network problems etc.)

c

c

b

abrachytherapy

Effects more relevant to “fast” use cases

Maria Grazia Pia

Effects of slow Effects of slow worker nodesworker nodes

More visible when few worker nodes

Similar execution time of all tasks on

all nodes

One slow node doubles the

total simulation duration

Both simulations executed in the same environment in

identicalComputing Elements

Possible explanation: variable workload on the

same nodes

Maria Grazia Pia

““Fast” use case: brachytherapyFast” use case: brachytherapy

Sequential

1 task = 25 ± 0.5 CPU s

Total simulation = 417 ± 8 min on lcgui003

Task duration

Total simulation duration

~ factor 6 between min

and max duration

~ factor 4 between min

and max duration

40 DIANE workers

On the grid

64% runs terminated < 30 min96% runs terminated < 40 min

Ideal expectation with 40 workers: ~10 min for the whole simulation

Maria Grazia Pia

More computationally intensive use More computationally intensive use case: hadrontherapycase: hadrontherapy

Sequential

1 task = 50’ ± 50 s CPU time

Total simulation = 16.7 h ± 17 min

20 DIANE workers

On the grid

84% runs terminated < 100 min

Ideal expectation with 20 workers: ~50 min for the whole simulation

Maria Grazia Pia

Large scale medical Large scale medical simulationssimulations

Test case: hadrontherapy– Increase statistics by 2 orders of magnitude– 10M events for high precision physics studies– Expected duration of whole production: 2-3 days

3 large statistics runs– 2 did not complete (80% done)– 1 terminated successfully

Medical linac– Even more demanding than high statistics hadrontherapy– “Low” statistics demonstration OK– Realistic scale stable production does not look practical yet

Maria Grazia Pia

ConclusionConclusion

Politically correct

Grid technology offers unprecedented opportunities to the medical physics community for fast,

high precision Monte Carlo simulation

Politically incorrect

Grid technology is not mature yet for stable operation in a clinical

environment by non-expert users

Scientific

Both statements above reflect reality

Quantitative tests evaluate the extent of success and contribute to identify

problems and bottlenecks

Measurements in real-life use cases are the path to solve problems

Maria Grazia Pia

Anton Lechner

Austrian Doctoral StudentCERN IT/PSS

A BIG thank you!

Jürgen Knobloch Patricia Mendez Lorenzo Kuba Moscicki

Unfortunately not present at CHEP to get the credit he deserves!

and

Hurng-Chung Lee

Alberto Ribon

Alfonso ManteroGraduate Student at

INFN Genova1st Geant4-DIANE project

CERN IT/PSS

Maria Grazia Pia

IEEE Transactions on Nuclear ScienceIEEE Transactions on Nuclear Sciencehttp://ieeexplore.ieee.org/xpl/RecentIssue.jsp?puNumber=23http://ieeexplore.ieee.org/xpl/RecentIssue.jsp?puNumber=23

Prime journal on technology in particle/nuclear physics

Rigorous review process Associate Editor dedicated to computing papers

Various papers associated to CHEP 2006 published on IEEE TNS

Computing-related papers are welcomeComputing-related papers are welcome

Manuscript submission: http://tns-ieee.manuscriptcentral.com/

Publications on refereed journals are beneficial not only to authors, but to the whole community of computing-oriented physicists

Our “hardware colleagues” have better established publication habits…

Further info: Maria.Grazia.Pia@cern.ch