out-of core streamline generation using flow-guided file layout
DESCRIPTION
Out-of core Streamline Generation Using Flow-guided File Layout. Chun-Ming Chen 788 Project. Background. Visualize flow fields with streamlines Scientific data is huge Traditional: Compute in clusters Drawbacks: High Equipment Cost Inter-node communication. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Out-of core Streamline Generation Using Flow-guided File Layout](https://reader035.vdocument.in/reader035/viewer/2022062323/56816224550346895dd253c2/html5/thumbnails/1.jpg)
1
Out-of core Streamline Generation Using Flow-guided File Layout
Chun-Ming Chen
788 Project
![Page 2: Out-of core Streamline Generation Using Flow-guided File Layout](https://reader035.vdocument.in/reader035/viewer/2022062323/56816224550346895dd253c2/html5/thumbnails/2.jpg)
2
Background
• Visualize flow fields with streamlines • Scientific data is huge – Traditional: Compute in clusters– Drawbacks: • High Equipment Cost• Inter-node communication
http://vidi.cs.ucdavis.edu/publicationshttp://nuweb2.neu.edu/
![Page 3: Out-of core Streamline Generation Using Flow-guided File Layout](https://reader035.vdocument.in/reader035/viewer/2022062323/56816224550346895dd253c2/html5/thumbnails/3.jpg)
3
Background
• Nowadays: multi-core CPU on single machine• May not have enough memory capacity• Out-of-core computation is needed– Out-of-core: data cannot be fully loaded into main
memory
![Page 4: Out-of core Streamline Generation Using Flow-guided File Layout](https://reader035.vdocument.in/reader035/viewer/2022062323/56816224550346895dd253c2/html5/thumbnails/4.jpg)
4
Goal
Compute streamlines on a lower-cost multi-core machine with limited memory, given arbitrary seeds
![Page 5: Out-of core Streamline Generation Using Flow-guided File Layout](https://reader035.vdocument.in/reader035/viewer/2022062323/56816224550346895dd253c2/html5/thumbnails/5.jpg)
5
Demand Paging Algorithm
• Preparation Stage:– Break flow fields into blocks
• Streamline Generation Stage:– Only load needed blocks during computation– Release least recently used (LRU) block when
memory full
Load data from Disk Compute Release data
(LRU)
Store data in memory
pool
![Page 6: Out-of core Streamline Generation Using Flow-guided File Layout](https://reader035.vdocument.in/reader035/viewer/2022062323/56816224550346895dd253c2/html5/thumbnails/6.jpg)
6
Multi-core streamline computation
Threaded Computation
Seeds for block 1
Seeds for block 2
Seeds for block 3
Seeds for block 4
Threaded Computation
New seeds generated from
block 1
Job Queue
![Page 7: Out-of core Streamline Generation Using Flow-guided File Layout](https://reader035.vdocument.in/reader035/viewer/2022062323/56816224550346895dd253c2/html5/thumbnails/7.jpg)
7
Problem of Out-of-core Computation
• Earlier tests: 1Gb Data– Environment:• 8-core Intel Machine• Limit 25Mb memory usage
– Time Generating streamlines: 8.436 s– Time Loading flow field : 32.524 s
• IO is the bottle neck
![Page 8: Out-of core Streamline Generation Using Flow-guided File Layout](https://reader035.vdocument.in/reader035/viewer/2022062323/56816224550346895dd253c2/html5/thumbnails/8.jpg)
8
More tests
• Read all blocks in a 6Gb data• Unit block size: float 16x16x16 (49152 bytes)
Total 131,072 blocks– Random access: 249.146794 sec– Sequential read: 13.594309
sec– Reverse-Sequential read: 264.262752 sec
• Sequential read can be 20 times faster• Reason: Disk Prefetching
![Page 9: Out-of core Streamline Generation Using Flow-guided File Layout](https://reader035.vdocument.in/reader035/viewer/2022062323/56816224550346895dd253c2/html5/thumbnails/9.jpg)
9
File Layout
• Re-arrange data to increase more sequential reads
• Hilbert Curve Layout:
![Page 10: Out-of core Streamline Generation Using Flow-guided File Layout](https://reader035.vdocument.in/reader035/viewer/2022062323/56816224550346895dd253c2/html5/thumbnails/10.jpg)
10
Result of Scheduling for Hilbert Curve Layout
• Scheduler: only read forward• Test: 1Gb Data– Environment:
• 8-core Intel Machine• Limit 25Mb memory usage
• Old test:– Time Generating streamlines: 8.436 s– Time Loading flow field : 32.524 s
• Hilbert layout:– Time Generating streamlines: 7.890 s– Time Loading flow field : 0.610 s
![Page 11: Out-of core Streamline Generation Using Flow-guided File Layout](https://reader035.vdocument.in/reader035/viewer/2022062323/56816224550346895dd253c2/html5/thumbnails/11.jpg)
11
Layout By Flow Direction
![Page 12: Out-of core Streamline Generation Using Flow-guided File Layout](https://reader035.vdocument.in/reader035/viewer/2022062323/56816224550346895dd253c2/html5/thumbnails/12.jpg)
12
Next And Conclusion
• Next:– Better layout?– Re-arrange data based on flow direction– NP-hard Problem
• Conclusion:– If we want to analyze large scientific data in a single
machine, out-of-core computation is required now and also in the future
– Good File layout is important for out-of-core computation