Download - LBNL VACET Activities Hank Childs Computer Systems Engineer - Visualization Group August 24, 2009
LBNL VACET Activities
Hank ChildsComputer Systems Engineer - Visualization Group
August 24, 2009
Director’s Review of CRD | August 24 - 26, 2009
Outline• Overview: VACET & VisIt
• Deployment & Outreach: math infrastructure
• Research: QDV & parallel streamlines
• Preparing for the Future: hero runs & VisIt infrastructure
Director’s Review of CRD | August 24 - 26, 2009
Outline• Overview: VACET & VisIt
• Deployment & Outreach: math infrastructure
• Research: QDV & parallel streamlines
• Preparing for the Future: hero runs & VisIt infrastructure
Director’s Review of CRD | August 24 - 26, 2009
VACET Mission and Vision
• Mission: enable scientific insight for petascale data.• Strategy: Leverage sci-vis and analytics software
technology as an enabling technology. • Vision: adapt, extend, create, and deploy data
understanding technologies for science stakeholders• Why are we a center?
– As a center, well positioned to respond to diverse needs/objectives through coordinated R&D, software engineering, outreach efforts.
Director’s Review of CRD | August 24 - 26, 2009
VACET Organization
• Teams: stakeholder projects, R&D projects, software engineering projects.
• Executive committee: cross-institutional, cross-team coordination.
PIsBethel/LBNL, Johnson/Utah
Executive CommitteeBethel/LBNL, Brugger/LLNL, Johnson/Utah, Joy/UCD, Ahern/ORNL
+ Pascucci/Utah, Childs/LBNL
R&D TeamsStakeholder Projects Software Engineering Teams (per site)
Chief Software Engineer, Childs/LBNL
Director’s Review of CRD | August 24 - 26, 2009
VisIt is a richly featured, turnkey application for large data.
• VisIt is an open source, end user visualization and analysis tool for simulated and experimental data– Used by: physicists, engineers,
code developers, vis experts– >100K downloads on web
• R&D 100 award in 2005• Used “heavily to exclusively” on 8 of
world’s top 12 supercomputers
27B elementRayleigh-Taylor Instability(MIRANDA, BG/L)
Director’s Review of CRD | August 24 - 26, 2009
VisIt has a rich feature set that can impact many science areas.
• Meshes: rectilinear, curvilinear, unstructured, point, AMR• Data: scalar, vector, tensor, material, species• Dimension: 1D, 2D, 3D, time varying• Rendering (~15): pseudocolor, volume rendering,
hedgehogs, glyphs, mesh lines, etc…• Data manipulation (~40): slicing, contouring, clipping,
thresholding, restrict to box, reflect, project, revolve, …• File formats (~85)• Derived quantities: >100 interoperable building blocks• Many general features: position lights, make movie, etc• Queries (~50): ways to pull out quantitative information,
debugging, comparative analysis
Director’s Review of CRD | August 24 - 26, 2009
VisIt and VACET provide significant mutual leverage.
• VisIt represented over 50 person-years of effort at the time of VACET being funded
– VisIt contains over one million lines of code• Eric Brugger, NNSA VisIt lead: “VACET has assumed the lead
role in ensuring that VisIt will be able to handle the largest data sets.”
2004-6
User communitygrows, includingAWE & ASC Alliance schools
Fall ‘06
VACET is funded
Spring ‘08
AWE enters repo
2003
LLNL user communitytransitioned to VisIt
2005
2005 R&D100
2007
SciDAC Outreach Center enablesPublic SW repo
2007
Saudi Aramcofunds LLNL to support VisIt
Spring ‘07
GNEP funds LLNL to support GNEP codes at Argonne
Summer‘07
Developers from LLNL, LBL, & ORNLStart dev in repo
‘07-’08
UC Davis & UUtah research done in VisIt repo
2000
Project started
‘07-’08
Partnership withCEA is developed
2008
Institutional supportleverages effort from many labs
Spring ‘09
More developersEntering repo allthe time
Director’s Review of CRD | August 24 - 26, 2009
Outline• Overview: VACET & VisIt
• Deployment & Outreach: math infrastructure
• Research: QDV & parallel streamlines
• Preparing for the Future: hero runs & VisIt infrastructure
Director’s Review of CRD | August 24 - 26, 2009
We have made an impact for APDEC• PI: Phil Colella (LBNL), SciDAC Applied
Partial Differential Equations Center• Accomplishment(s)
– Software engineering to “bring product to market”. Performance improvements, interface enhancements, file readers, visual data exploration techniques.
• Science Impact– Visualization on large AMR data sets, and
using parallel platforms.– Trail blazed for future VACET work.
Director’s Review of CRD | August 24 - 26, 2009
VACET has improved VisIt’s AMR capabilities significantly.
• Improvements:– Optimized many, many
routines, resulting in end-to-end speedups of >10X
– Added critical debugging functionality, such as spreadsheets and the ability to connect with a debugger.
– Mapped AMR grids
- Macros and python callbacks for customizability.
- Responded to over 30 requests for interface changes and bug fixes.
Director’s Review of CRD | August 24 - 26, 2009
VACET’s efforts have led to VisIt being adopted by important ASCR stakeholders.
+ see our review document for 9 more letters of support from SciDAC-funded
groups using VisIt
Director’s Review of CRD | August 24 - 26, 2009
Outline• Overview: VACET & VisIt.
• Deployment & Outreach: math infrastructure.
• Research: QDV & parallel streamlines
• Preparing for the Future: hero runs & VisIt infrastructure
Director’s Review of CRD | August 24 - 26, 2009
QDV and Accelerator Modeling• Accomplishment:
– Algorithms + deployment in VisIt to perform interactive visual data analysis (identify, track, analyze beam particles) in multi-TB simulation data.
• Science Impact:– Replace serial process that took hours with
one that takes seconds.– New capability: rapid data exploration and
analysis.• SC08 paper• Customer: C. Geddes (LBNL)/COMPASS• Collaborators: SciDAC SDM Center (FastBit)
Tech-X (Accelerator scientists)
Director’s Review of CRD | August 24 - 26, 2009
Streamlines represent an activity spanning research to SWE to data insight.
Stakeholders:“VisIt is bad at
streamlines”
Stakeholders:“VisIt is bad at
streamlines”
“We need parallel streamlines” / “Wow! Parallel streamlines
is hard”
“We need parallel streamlines” / “Wow! Parallel streamlines
is hard”
Deployment effort in VisIt by SEG
Research effort:Efficient parallel
streamline generation
This work has had a broad and varied impact.
Used in analysis by SHOCKS center (image
courtesy SHOCKS website)
Used in analysis by SHOCKS center (image
courtesy SHOCKS website)
Poincare analysis for fusion community
Poincare analysis for fusion community
Streamlines used in visualization of type II
supernova collapse
Streamlines used in visualization of type II
supernova collapseResearch vehicle (and delivery vehicle) on streamline generation of AMR data sets.
Research vehicle (and delivery vehicle) on streamline generation of AMR data sets.
Level 0 only (incorrect)
Correctly traversing all AMR levels
217 pin reactorcooling simulation.Run on ¼ of Argonne BG/P.
Director’s Review of CRD | August 24 - 26, 2009
This work has had a broad and varied impact.
“First light” picture showing mixing of cold and warm air inlets. (Nek5000/Fischer, Argonne, INCITE run)
“First light” picture showing mixing of cold and warm air inlets. (Nek5000/Fischer, Argonne, INCITE run)
Director’s Review of CRD | August 24 - 26, 2009
Existing parallel integral curve techniques have suffered from load imbalance.
Two extremes:• Partition data over processors
and pass particles amongst processors Parallel inefficiency!
• Partition seed points over processors and process necessary data for advection Redundant I/O!
Notional streamlineexample
Notional streamlineexample
P0 P0 P0 P0 P0
P1 P1 P1 P1 P1
P2 P2 P2 P2 P2
P3 P3 P3 P3 P3
P4 P4 P4 P4 P4
P0
P1P2
P3P4
Director’s Review of CRD | August 24 - 26, 2009
We have greatly advanced the state of the art for parallel integral curve techniques. Hybrid solution:• Master-slave approach that
adapts between parallel inefficiencies and redundant I/O
P0P0
P1
P1P2
P2P3
P4
Iteration Action
0 P0 reads B0,P3 reads B1
1 P1 passes points to P0,P4 passes points to P3,P2 reads B0
0: Read
0: Read
Notional streamlineexample
Notional streamlineexample
1: Pass
1: Pass1: Read
- Decision of when to pass or read is simplified here and based on load of processors owning data. - Heuristic weights against I/O, but allows for redundant I/O- All coordination done by slave masters.
Director’s Review of CRD | August 24 - 26, 2009
Outline• Overview: VACET & VisIt
• Deployment & Outreach: math infrastructure
• Research: QDV & parallel streamlines
• Preparing for the Future: hero runs & VisIt infrastructure
Director’s Review of CRD | August 24 - 26, 2009
Trillion-cell experiment: objectives• Research questions:
– Is it possible/feasible to run production-quality visual data analysis s/w on large machines and on large data sets?
– What obstacles/bottlenecks do we encounter at this level of scale?
• Why?– VACET’s job: provide production-quality tools for
large data. • This is demonstrating that these tools are
viable.
Director’s Review of CRD | August 24 - 26, 2009
Trillion cell experiment methodology:
What tools, what machines, what data?• Application: VisIt 1.11.1
– VisIt is the first and only visualization application that is part of the “Joule” metric
• Machines: Cray (franklin/LBL, jaguar/ORNL), BG/P (dawn/LLNL), AIX (purple/LLNL), Linux (Juno/LLNL), Sun Linux (ranger/UTexas)
• Data: – 5123 sample data, astrophysics simulation (courtesy T.
Mezzacappa, B. Messer, S. Bruenn, R. Budjiara)– Upsampled to target problem size– Data sizes ranged to 1T (for 16K cores) to 4T (for 64K
cores), with 0.5T/8K for AIX/Purple
Director’s Review of CRD | August 24 - 26, 2009
Trillion cell experiment methodology:What techniques?
• Two common visualization techniques:– Isocontouring, volume rendering
2T zones, 32K procs on Jaguar
2T zones, 32K procs on Franklin
Director’s Review of CRD | August 24 - 26, 2009
Trillion cell experiment “speed bumps”
• Application bottleneck: many-to-one status update
– Workaround for study, fix to appear in future release of VisIt
• Application optimization: an NxN buffer for communication, which works well at small scale, proved problematic at large scale.
– Workaround: remove optimization, evaluate best course of action for future VisIt release.
• System issues– Loading shared libraries takes a long time, as much as 5
minutes. Potential fix: implement VisIt plugins as static libraries.
Director’s Review of CRD | August 24 - 26, 2009
Trillion cell experiment: Significance of Results
• Production tool (VisIt) runs effectively at high concurrency, on large problem sizes, and on diverse platforms.– Test of visual data analysis infrastructure– Infrastructure is the underpinning of many different
algorithms, not just isocontouring or volume rendering.
• Minor fixes to become part of production VisIt release (later this year).
• Successfully processed “tomorrow’s data today.”
~1980 ~2000 ~2020
1M 1B 1T
We are here
Director’s Review of CRD | August 24 - 26, 2009
VACET has contributed more than just algorithms and research to VisIt.
• Production software: bug tracking, regression testing, regular releases
• Through our efforts, VisIt now feels much more like an open source project:– Transitioned software repository to NERSC; now
accessible to a large community.• ~30 developers with write access from 10
institutions– Public mailing lists, archived and searchable.
• ~300 subscribers, get ~300 posts per month– Wiki pages on usage and development.
Director’s Review of CRD | August 24 - 26, 2009
Summary
• The LBNL portion of VACET has made a huge impact in the ASCR community– In deployment, in research, and in paving the way for
the future• VACET is partnering with many researchers in the ASCR
community.• VisIt, the production visualization tool for VACET, is being
developed by a vibrant community and has demonstrated applicability at the petascale.
• LBNL is a major center of gravity for the VisIt project.