laurent castanié (earth decision / paradigm – inria lorraine – project alice) christophe mion...
TRANSCRIPT
Laurent Castanié(Earth Decision / Paradigm – INRIA Lorraine – Project ALICE)
Christophe Mion(INRIA Lorraine – Project ALICE)
Xavier Cavin(INRIA Lorraine – Project ALICE)
Bruno Lévy(INRIA Lorraine – Project ALICE)
Distributed Shared Memory for Roaming Large Volumes
Distributed Shared Memory for Roaming Large Volumes
2
OutlineOutline
Introduction• Large volumes in the Oil and Gas EP domain• Previous work: single-workstation cache system
COTS cluster solution: DHCS• Distributed volume rendering• Distributed data management
Real-time roaming in gigantic data sets with DHCS Conclusions
3
OutlineOutline
Introduction• Large volumes in the Oil and Gas EP domain• Previous work: single-workstation cache system
COTS cluster solution: DHCS• Distributed volume rendering• Distributed data management
Real-time roaming in gigantic data sets with DHCS Conclusions
4
Reservoir scale volume300x400x400 (~50 MB)100-300 Km2
Targeted ROI1000x1000x1000 (~1 GB)
Targeted volume4000x5000x5000 (~100 GB)
Reservoir scale ROI100x200x200 (~4 MB)
Targeted ROI~ 250x typical
reservoir scale ROI
REGIONAL SCALE10000-30000 Km2
IntroductionInterpretation scales in Oil and Gas EPIntroductionInterpretation scales in Oil and Gas EP
5
IntroductionOOC visualization on a single workstationIntroductionOOC visualization on a single workstation
8% 6%
0.5%
100 GB
Data Volume(Disk)
512 MB
Graphics Card(V-RAM)
8 GB
Workstation(RAM)
6
IntroductionOOC visualization on a single workstationIntroductionOOC visualization on a single workstation
Bhaniramka and Demange, IEEE VolVis 2002, OpenGL Volumizer
Plate et al., VISSYM 2002, Octreemizer
Castanie et al., IEEE Visualization 2005VolumeExplorer (coupling OOC visualization and data processing)
Probe-based roaming systems with LRU volume
paging
7
Efficient solution up to 20-30 GB, however:
ROI size is limited to the amount of graphics memory available; Performance decreases rapidly when the size of data on disk
increases over 30 GB.
=> How to SCALE our solution up to 100-200 GB ?
Distributed Hierarchical Cache System (DHCS)on COTS cluster
IntroductionOOC visualization on a single workstationIntroductionOOC visualization on a single workstation
8
OutlineOutline
Introduction• Large volumes in the Oil and Gas EP domain• Previous work: single-workstation cache system
COTS cluster solution: DHCS• Distributed volume rendering• Distributed data management
Real-time roaming in gigantic data sets with DHCS Conclusions
9
SLAVE SLAVE NODENODE
SLAVE SLAVE NODENODE
SLAVE SLAVE NODENODE
SLAVE SLAVE NODENODE
SLAVE SLAVE NODENODE
SLAVE SLAVE NODENODE
SLAVE SLAVE NODENODE
SLAVE SLAVE NODENODE
MASTER MASTER NODENODE
1. Segmentation
2. Distribution
3. Rendering
4. Composition
Distributed volume renderingSort-last parallel volume renderingDistributed volume renderingSort-last parallel volume rendering
10
Distributed volume renderingPipelined binary-swap compositingDistributed volume renderingPipelined binary-swap compositing
P0 P1 P2 P3
Result on MASTER
node
Ma et al., IEEE CG&A 1994Binary-swap compositing
Cavin et al., IEEE Visualization 2005Cavin et al., Eurographics PGV 2006DViz pipelined implementation
11
ROI size several GBs => ~15-20 fps
One virtual graphics
card
8 GB
DViz
Master
Distributed volume renderingPipelined binary-swap compositingDistributed volume renderingPipelined binary-swap compositing
16 nodes with GeForce 6800 ULTRA - 512 MB
12
OutlineOutline
Introduction• Large volumes in the Oil and Gas EP domain• Previous work: single-workstation cache system
COTS cluster solution: DHCS• Distributed volume rendering• Distributed data management
Real-time roaming in gigantic data sets with DHCS Conclusions
13
Very low disk to memory bandwidth Faster transfers through the network?
?
Distributed data managementLimited disk to memory bandwidthDistributed data managementLimited disk to memory bandwidth
Graphics Card(V-RAM)
Workstation(RAM)
Data Volume(Disk)
Gigabit EthernetNetwork
~1 GB/s~50 MB/s
14
0 100 200 300 400 500 600
Bandwidth (MB/s)
1xGigabit Ethernet2xGigabit EthernetInfiniband 4xSATA 150SCSI U320 2x RAID0 Maxtor AtlasSATA 150 2x RAID0 WD Raptor
4x faster transfers through the network
50 MB/s
Network
Disk
220 MB/s
Distributed data managementDisk Vs Network bandwidthDistributed data managementDisk Vs Network bandwidth
120 MB/s
500 MB/s
15
Fully dynamic memory state that must be kept up-to-date
~50 MB/s
~220 MB/s1 1
2
Distributed data managementOur fully dynamic implementation (DHCS)Distributed data managementOur fully dynamic implementation (DHCS)
8 GB8 GB8 GB8 GB
16
OutlineOutline
Introduction• Large volumes in the Oil and Gas EP domain• Previous work: single-workstation cache system
COTS cluster solution: DHCS• Distributed volume rendering• Distributed data management
Real-time roaming in gigantic data sets with DHCS Conclusions
17
30 copies of the Visible Human data set= 5580x5400x3840 ~ 107 GB
ROI 1000x1000x1000 ~ 1 GB
=> Real-time rendering and volume roaming at full resolution at 12 fps on average on a 16-node cluster
ResultsReal-time rendering and volume roamingResultsReal-time rendering and volume roaming
Full resolution volume roaming
Full resolution volume rendering
18
OutlineOutline
Introduction• Large volumes in the Oil and Gas EP domain• Previous work: single-workstation cache system
COTS cluster solution: DHCS• Distributed volume rendering• Distributed data management
Real-time roaming in gigantic data sets with DHCS Conclusions
19
Volume visualization of ~100 GB Volume roaming a ROI of
several GBs Cluster-based hierarchical cache
system• Distributed volume rendering• Distributed data management
Compression techniques Better load balancing of the communications on the network Pre-fetching strategies to hide disk access Other use cases:
• Combination of multiple attributes of 100 GB each• Real-time full resolution volume slicing at 20-30 slices per second
ConclusionsConclusions
20
AcknowledgementsAcknowledgements
This work involved and has been supported by:• Earth Decision (now part of Paradigm) http://
www.earthdecision.com• LORIA / INRIA – Project ALICE http://alice.loria.fr• DViz http://www.dviz.fr• Region Lorraine (CRVHP) • GOCAD consortium http://www.gocad.org