1 peter sloot: computational science, university of amsterdam, the netherlands. interactive problem...

1Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Interactive Problem Solving: The Polder Meta Computing Inititiative

Peter Sloot

Computational Science

University of Amsterdam, The Netherlands

2Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Ariadne’s Red-Rope

– From PSE to Virtual Laboratory and Motivation

– Architecture• Infrastructure• Job Level: Hierarchical Scheduling• Resource management: Task-migration

– Interaction && Case implementation

– Interactive Algorithms

3Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Virtual Laboratory Environment

Internet 2 Wide Area Network

ViSE

Net Client App. User MRI/CT

Distributed Computing & Gigabit Local Area Network

Local User

Local User

Physical apparatus

Virtual-lab Information Management for Cooperation (VIMCO)

Communication & collaboration (ComCol)

Virtual Simulation & Exploration Environment (ViSE)

Advanced Scientific Domains

Computational Physics

System Engineering

Computational Bio-medicine

4Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Interactive Computing: Why?– Goal: From Data, via Information to Knowledge

– Complexity: Huge data-sets, complex processes

– Approach: Parametric exploration and sensitivity analyses:• Combine raw (sensory) data with simulation• Person in the loop:

• Sensory interaction • Intelligent short-cuts

5Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Intro: Case study from biomedicine...

6Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Changing the Paradigm

In Vivo

In Vitro

In Silico

7Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Changing the Paradigm

In Vivo

In Vitro

In Silico

8Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Changing the Paradigm

In Vivo

In Vitro

In Silico

9Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Current Situation

Observation

Diagnosis & Planning

Treatment

10Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

New Possibilities in the VL

Fast, High-throughputLow Latency

Internet

High Performance Super Computing

• Time and Space Independence

• 3D Information

• Simulation based planning

• Surgeon ‘in the loop’

11Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Experimental set-up

12Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Architecture

13Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Architecture Continued: Hybrid system

– Host: The DAS• 24 node parallel cluster in a

200 node wide area machine

• 200 MHz Pentium Pro

• Myrinet 150MB/s

• ATM wide-area interconnect between clusters

9 10 11

8

7

6

141312

15

16

17

2 1 0

3

4

5

212223

20

19

18

GRAPE1 GRAPE0

ATM

Origine 2000

Cave

14Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Immersive Environments

15Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

3D Information and Interaction

16Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Problem: Curse of dynamics:

Static task load Dynamic task load

Static resource load Dynamic resource load

Static task allocation Predictable

reallocationDynamical reallocation

17Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Solution To Curse

– Performance of a parallel program usuallydictated by slowest task• Task resource requirements and available resources

both vary dynamically• Therefore, optimal task allocation changes• Gain must exceed cost of migration

– Resources used by long-running programs may be reclaimed by owner

18Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Dynamite Initial State

Two PVM tasks communicating through a network of daemonsMigrate task 2 to node B

Node A Node B

Node C

PVMDA

PVMDB

PVMDC

PVMtask 1

PVMtask 2

19Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Prepare for Migration

Create new context for task 2Tell PVM daemon B to expect messages for task 2Update routing tables in daemons (first B, then A, later C)

Node A Node B

Node C

PVMDA

PVMDB

PVMDC

PVMtask 1

Program

PVM

Ckpt

Newcontext

20Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Checkpointing

Send checkpoint signal to task 2Flush connectionsCheckpoint task to disk

Node A Node B

Node C

PVMDA

PVMDB

PVMDC

PVMtask 1

Program

PVM

Ckpt

Newcontext

21Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Cross-cluster checkpointing(design)

Send checkpoint signal to task 2Flush connections, close filesCheckpoint task to disk via helper task

Node A Node B

Node C

PVMDA

PVMDB

PVMDC

PVMtask 1

Program

PVM

Ckpt

Helpertask

22Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Restart Execution

Restart checkpointed task 2 on node BResume communicationsRe-open & re-position files

Node A Node B

Node C

PVMDA

PVMDB

PVMDC

PVMtask 1

NewPVMtask 2

23Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Special considerations

– Preserve communication• PVM should continue to run as if nothing happened• Use location independent addressing

– Open files• Preserve open file state

24Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Performance

– Migration speed largely dependent on the speed of shared file system• and that depends mostly on the network

– NFS over 100 Mbps Ethernet• 0.4 s < Tmig < 15 s for

2 MB < sizeimg < 64 MB

– Communication speed reduced due to added overhead• 25% for 1 byte direct messages

• 2% for 100 KB indirect messages

25Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Current status: Dynamite Part

– Checkpointer operational under• Solaris 2.5.1 and higher (UltraSparc, 32 bit)

• Linux/i386 2.0 and 2.2 (libc5 and glibc 2.0)

– PVM 3.3.x applications supported and tested• Pam-Crash (ESI) - car crash simulations• CEM3D (ESI) - electro-magnetics code• Grail (UvA) - large, simple FEM code• NAS parallel benchmarks• BloodFlow

– MPI and socket (Univ. of Krakow) libraries available– Scheduling not yet satisfactory

26Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Architecture: Revisited

27Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Design Considerations

– High Quality presentation

– High Frame rate

– Intuitive interaction

– Real-time response

– Interactive Algorithms

– High performance computing and networking...

28Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Problem: Time, time what has become of us?

29Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Solution: Asynchronicity

30Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

A police officer to guide the asynchronous processes

31Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Runtime Support

– Need generic framework to support modalities

– Need interoperability

– High Level Architecture (HLA):• data distribution across heterogeneous platforms• flexible attribute and ownership mechanisms• advanced time management

32Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Provoking a bit…

Progress in natural sciences comes from taking things apart ...

Progress in computer science comes from bringing things together...

33Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Proof is in the pudding...

– Diagnostic Findings

• Occluded right iliac artery

• 75% stenosis in left iliac artery

• Occluded left SFA

• Diffuse disease in right SFA

34Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Problem: From Image to Simulation

MR Scan of Abdomen MR Scan of Legs

35Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Solution: 3DManual initialization

Place start point

Wave propagates from start- to end point

Place one or more end points

Backtrack = first estimation of the

centerline

Distance Transform from vessel wall to center centerline

Wave propagates from ‘centerline’ vessel

wall

36Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Wavefront PropagationPlace start point

Wave propagates from start- to end point

Place one or more end points

Backtrack = first estimation of the

centerline

Distance Transform from vessel wall to center centerline

Wave propagates from ‘centerline’ vessel

wall

37Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

MRA: BacktrackPlace start point

Wave propagates from start- to end point

Place one or more end points

Backtrack = first estimation of the

centerline

Distance Transform from vessel wall to center centerline

Wave propagates from ‘centerline’ vessel

wall

38Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

MRA: Wavefront PropagationPlace start point

Wave propagates from start- to end point

Place one or more end points

Backtrack = first estimation of the

centerline

Distance Transform from vessel wall to center centerline

Wave propagates from ‘centerline’ vessel

wall

39Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

MRA: Distance TransformPlace start point

Wave propagates from start- to end point

Place one or more end points

Backtrack = first estimation of the

centerline

Distance Transform from vessel wall to center centerline

Wave propagates from ‘centerline’ vessel

wall

40Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

3-D selection of region of interest

41Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Tracking the vessels

42Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Building the Geometric Models

43Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

VR-Interaction

44Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Alternate Treatments

Angio w/ Fem-Fem &

Fem-Pop

AFB w/ E-S Prox.

Anast.

Angio w/Fem-Fem

AFB w/ E-E Prox.

Anast.

Preop

45Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Problem: Flow through complex geometry

– After determining the vascular structure simulate the blood-flow and pressure drop…

– Conventional CFD methods might fail:• Complex geometry• Numerical instability wrt interaction• Inefficient shear-stress calculation

46Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Solution to interactive flow simulation

– Use Cellular Automata as a mesoscopic model system:• Simple local interaction• Support for real physics and heuristics• Computational efficient

47Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Mesoscopic Fluid Model

– Fluid model with Cellular Automata rules

– Collision: particles reshuffle velocities

– Imposed Constraints• Conservation of mass• Conservation of momentum• Isotropy

Details...

48Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

...Equivalence with NS– For lattice with enough symmetry: equivalent to the continuous incompressible Navier-Stokes equations:

uuu

u

2 1

0

Pt

u

Implicit parallel and complex geometry support.

49Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Efficient Calculation of Shear-Stress

AND the momentum stress tensor that is linearly related to the shear stresses

i

if i

iif cu

i

iiif ccΠ

x

u

x

u

~

Perpendicular momentum transfer:

From LBE scheme:

50Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Velocity Magnitude

10 cm/sec

0 cm/sec

51Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Peak Systolic Pressures - Rest

150 mmHg

50 mmHg

Angio w/ Fem-Fem &

Fem-Pop

AFB w/ E-S Prox.

Anast.

Angio w/Fem-Fem

AFB w/ E-E Prox.

Anast.

Preop

52Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

… last slides...

53Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Internet and Web Software

Distributed Computer infrastructure

Central-part Virtual Laboratory

Physical Apparatus

User

ViSE ComCol VIMCO

Internet and Web Software

Distributed Computer infrastructure

Central-part Virtual Laboratory

User

Computing in Physics

VL for Material Science

ViSE ComCol VIMCO

Meta data Integration

Computing in Engineering

Traffic Payment for mobility

Combining problem solving & data intensive

environments

Bio-medicalComputation

Study of blood flow through

veins

Integration of simulation &

visualization by man in the loop

Bio- informaticsEnvironment

DNA Research

Combing datamining & intelligent data bases

Cultural Inheritance

Environment

Art objects preservationrestoration

Collaborative data

integration

Computing in Engineering

Apply VL in non-quality of service

environment

Modeling VL in non-QoS situation

environment

Other Virtual Laboratory Applications @ UvA

54Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

AcknowledgementsStanford:

Charley Taylor, PhD.

Christopher K. Zarins, PhD. M.D.

UvA:

Robert Belleman

Alfons Hoekstra, PhD

Dick van Albada, PhD

Benno Overeinder, PhD

Krakow

Marian Bubak, PhD

Kamil Iskra

RUL/AZL:

H. Reiber, PhD.

Bloem, PhD, M.D.

SARA:

A. de Koning, PhD.

Arcobel:

S. ten Den

IBM:

J. Geise

55Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Support

ICES-KIS-1

ICES-KIS-2

KNAW

NWO/FOM

IBM

SARA

SGI

Platform HPCN

56Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

http://science.uva.nl/~sloot

sloot@science.uva.nl

57Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

58Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

1955 1965 1975 1985 19950.1

1

10

100

1000

10.000

100.000

2005

MFlop/s

1000.000

IBM 704

CDC 6600

Cray X-MP

Cray Y-MP

CM-5

ASCI-RedASCI-Blue

2D Plasma

48 hr Weather

72 hr Weather

Pharmaceutical

?

Structural Biology

Oil reservoir

59Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Results - Mean Flow Rates (ml/min) - Rest

Artery Preop AFB/E-S AFB/E-E Angio/FF Angio/FF-FP

Abd. Aorta 3009 3460 3460 3460 3460

Aortic Bifurc. 670 307 15 1310 1385

R. Int. Iliac 134 156 212 144 132

L. Int. Iliac 464 148 150 335 360

R. Femoral 216 615 663 444 546

L. Femoral 464 550 781 747 618

R. Popliteal 30 283 309 162 303

L. Popliteal 272 351 362 290 277

60Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Cellular Automata

– 1966 Introduced by John von Neumann

– 1985 Stephen Wolfram suggested CA are capable of Universal Computation

– 1990 Lindgren et al., proved UC in 1D CA

61Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

1 0 1 0 0 1 1 0 0t=0

1 1 1 0 1 1 1 0 1t=1

0000

0011

0101

0111

1000

1011

1101

1110

Productie Regel 110

Time Evolution of 1D Cellular Automata 110

Back to Mesoscopic Models

63Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

The Lattice Gas model

– Fluid model with Cellular Automata rules

– Collision: particles reshuffle velocities

– Imposed Constraints• Conservation of mass• Conservation of momentum• Isotropy

tntntcn iiiii ,,1, xxx

64Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

The Hexagonal Lattice

65Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Collision rules examples

Two body collision N1 AND N4 => N2 AND N5 && N3 AND N6

Three body collision N2 AND N4 AND N6 => N1 AND N3 AND N5

Streaming and Colliding

67Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

From LGA to LBM

– Average LGA equation to get continuous values instead of boolean values

– Boltzmann molecular chaos assumption to factorize products in collision operator:

=> Iterate:33and iiiiii NNNNNf

),(),(1

),()1,( tftftftf eqiiiii xxxcx

68Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

From Micro Dynamics to Macro Dynamics (1)

– Taylor expansion to get continuous differential operators:

itiii

itiiiti

ff

fff

ee

e

221

221

69Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

From Micro Dynamics to Macro Dynamics (2)

– Chapman Enskog expansion of equilibrium Distribution Function:

– With imposed constraints:

221ii

eqii ffff

1,0 and 0

and

6

1

)(6

1

)(

6

1

6

1

jff

ff

ii

ji

i

ji

ii

eqi

i

eqi

e

eu

70Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

From Micro Dynamics to Macro Dynamics (3)

– Multi-scale expansion of time and space derivatives:

– Solve collision/flow equation for different order of

0

:balance)order nd-(2

.0

:balance)order st -(1

0021

1

Su rtrt

rt u

1

2 and 21

rttt

71Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Back to mesoscopic models