scientific visualization on gpu clusters
TRANSCRIPT
![Page 1: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/1.jpg)
PETER MESSMER, NVIDIA
SCIENTIFIC VISUALIZATION ON GPU CLUSTERS
![Page 2: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/2.jpg)
Visualization ≠ Rendering
![Page 3: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/3.jpg)
Visualization
Coordinate
transformations
Feature
extraction
Thresholding
Isosurfaces,
Isovolumes
Streamlines
Rendering Surface
Rendering
Volume
Rendering
Field Operators
(Gradient, Curl,.. )
Clip, Slice
Binning,
Resample Line
Rendering
Compositing
![Page 4: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/4.jpg)
Visualization
Feature
extraction
Isosurfaces,
Isovolumes
Streamlines
Field Operators
(Gradient, Curl,.. )
Filtering
Line
Rendering
Coordinate
transformations
Feature
extraction
Thresholding Isosurfaces,
Isovolumes
Streamlines
Field Operators
(Gradient, Curl,.. )
Clip, Slice
Binning,
Resample Rendering
Surface
Rendering
Volume
Rendering
Line
Rendering
Compositing
![Page 5: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/5.jpg)
Simulation: Data as needed in numerical algorithm
Filtering: Conversion of simulation data into data ready for rendering
Typical operations: binning, down/up-sampling, iso-surface extraction, interpolation, coordinate transformation, sub-selection, ..
Sometimes embedded in simulation
Rendering: Conversion of shapes to pixels (Fragment processing)
Compositing: Combination of independently generated pixels into final frame
Visualization
Filtering Rendering Compositing Simulation
THE VISUALIZATION PIPELINE
![Page 6: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/6.jpg)
PARTICULARITIES IN HPC VIZ
Images may be subject to copyright
Parallelism Remoteness Heterogeneity
![Page 7: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/7.jpg)
OUTLINE
Tools
Paraview, Visit, others
Rendering
Enable HW rendering on Tesla
Remote rendering
Compositing
Delivery
In-situ visualization
High-level overview. Some parts platform dependent. Check with your sysadmin.
![Page 8: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/8.jpg)
VISUALIZATION APPLICATIONS
![Page 9: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/9.jpg)
PARAVIEW Scalar, vector and tensor field data features
Plots: contour, curve, mesh, pseudocolor, volume,..
Operators: slice, iso-surface, threshold, binning,..
Quantitative and qualitative analysis/vis
Derived fields, dimension reduction, line-outs
Pick & query
Scalable architecture
Developed by Kitware, open source
http://www.paraview.org
![Page 10: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/10.jpg)
PARAVIEW’S SCALABLE ARCHITECTURE Client-server-server architecture
Server MPI parallel
Distributed filtering
GPU accelerated, parallel
rendering*
* requires X server on each node
![Page 11: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/11.jpg)
VISIT Scalar, vector and tensor field data features
Plots: contour, curve, mesh, pseudo-color, volume,..
Operators: slice, iso-surface, threshold, binning,..
Quantitative and qualitative analysis/vis
Derived fields, dimension reduction, line-outs
Pick & query
Scalable architecture
Open source
http://wci.llnl.gov/codes/visit/
![Page 12: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/12.jpg)
VISIT’S SCALABLE ARCHITECTURE
Client-server architecture
Server MPI parallel
Distributed filtering
(multi-)GPU accelerated,
parallel rendering*
* requires X server on each node
![Page 13: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/13.jpg)
SOME OTHER TOOLS
Wide range of visualization tools
Often emerged from specialized application domain
Tecplot, EnSight: structural analysis, CFD
IndeX: seismic data processing & visualization
IDL: image processing
Early adopters of visual programming
AVS/Express, OpenDX
![Page 14: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/14.jpg)
FURTHER READING
Paraview Tutorial:
http://www.paraview.org/Wiki/The_ParaView_Tutorial
VisIt Manuals/Tutorials:
http://wci.llnl.gov/codes/visit/manuals.html
![Page 15: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/15.jpg)
RENDERING
![Page 16: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/16.jpg)
OPENGL: API FOR GPU ACCELERATED RENDERING
• Primitives: points, lines, polygons
• Properties: colors, lighting, textures, ..
• View: camera position and perspective
• Shaders: Rendering to screen/framebuffer
• C-style functions, enums
See e.g. “What Every CUDA Programmer Should Know About OpenGL”
(http://www.nvidia.com/content/GTC/documents/1055_GTC09.pdf)
![Page 17: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/17.jpg)
A SIMPLE OPENGL EXAMPLE
glColor3f(1.0f,0,0);
glBegin(GL_QUADS);
glVertex3f(-1.0f, -1.0f, 0.0f); // The bottom left corner
glVertex3f(-1.0f, 1.0f, 0.0f); // The top left corner
glVertex3f(1.0f, 1.0f, 0.0f); // The top right corner
glVertex3f(1.0f, -1.0f, 0.0f); // The bottom right corner
glEnd();
glFlush();
State-based API
(sticky attributes)
Drawing
Render to screen
![Page 18: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/18.jpg)
glColor3f(1.0f,0,0);
glBegin(GL_QUADS);
glVertex3f(-1.0f, -1.0f, 0.0f);
glVertex3f(-1.0f, 1.0f, 0.0f);
glVertex3f(1.0f, 1.0f, 0.0f);
glVertex3f(1.0f, -1.0f, 0.0f);
glEnd();
glFlush();
float* vert={-1.0f, -1.0f, ..};
float* d_vert;
cudaMalloc(&d_vert, n);
cudaMemcpy(d_vert, vert, n,
cudaMemcpyHostToDevice);
renderQuad<<<N/128, N>>>(d_vert);
flushToScreen<<<..>>>();
?
![Page 19: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/19.jpg)
CUDA-OPENGL INTEROP: MAPPING MEMORY
• OpenGL: Opaque data buffer object (Mention here that this is mainly in “legacy” OpenGL and more control available in modern OpenGL)
• Vertex Buffer Object (VBO)
• User has very limited control
• CUDA: C-style memory management
• User has full control
• CUDA-OpenGL Interop:
Map/Unmap OpenGL buffers into CUDA memory space
cudaGraphicsGLRegisterBuffer(cuda_vbo, *vbo, flags);
cudaGraphicsMapResources(1, cuda_vbo, 0);
![Page 20: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/20.jpg)
CAN ALL GPUS SUPPORT OPENGL?
GeForce : standard feature set including OpenGL 4.5
Quadro : + certain highly accelerated features (e.g. CAD)
Tesla, K20X,m : Requires setting GPU Operation Mode (GOM) to “All on”
Tesla K40, K80: GOM “All on” by default
nvidia-smi –-query-gpu=gom.current
nvidia-smi –q
![Page 21: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/21.jpg)
OPENGL CONTEXT
State of an OpenGL instance
Incl. viewable surface
Interface to windowing system
Context creation: platform specific
Not part of OpenGL
Handled by Xserver in Linux/Unix-like systems
New alternative: EGL (currently: OpenGL ES only)
GLX: Interaction X<-> OpenGL
![Page 22: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/22.jpg)
Driver
GPU
3D X Server
Application
libGL
GLX OpenGL
libGLX Xlib
OPENGL ON HEADLESS SERVERS
libGLES
OpenGLES
Driver
GPU
3D X Server
Application
libGL
OpenGL
libEGL Xlib libGLES
OpenGL ES EGL
Current State Current State for OpenGL ES applications
![Page 23: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/23.jpg)
REMOTE RENDERING
![Page 24: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/24.jpg)
LOCAL RENDERING
Application
libGL Xlib
GLX
X11
Events
Com
mands
Driver
GPU, monitor attached
2D/3D X Server
OpenGL
![Page 25: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/25.jpg)
X-FORWARDING: THE SIMPLEST FORM OF “REMOTE” RENDERING
Application
libGL Xlib
OpenGL/GLX
X11 Events
X11 Commands
Driver
GPU, monitor attached
2D/3D X Server On remote system: export DISPLAY=59.151.136.110:0.0
Network
![Page 26: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/26.jpg)
SERVER-SIDE RENDERING + REMOTE VIZ APPLICATION
Driver
GPU
2D/3D X Server
Application
libGL Xlib
GLX
OpenGL
X11
Events X11
Cmds
Driver
GPU, monitor attached
Network
Client
![Page 27: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/27.jpg)
SERVER-SIDE RENDERING + SCRAPING
Driver
GPU
2D/3D X Server
Application
libGL Xlib
GLX
OpenGL
X11
Events X11
Cmds
Driver
GPU, monitor attached
Network
Scra
per
Images
Client
![Page 28: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/28.jpg)
CAPTURE/ENCODING WITH NVFBC/NVIFR
Driver
GPU
2D/3D X Server
Application
libGL Xlib
GLX
OpenGL
X11
Events X11
Cmds
Driver
GPU, monitor attached
Network
NV
FBC
/NV
IFR
Images
Client
H.264
https://developer.nvidia.com/grid-app-game-streaming
https://developer.nvidia.com/nvidia-video-codec-sdk
![Page 29: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/29.jpg)
GLX FORKING WITH INTERPOSER LIBRARY
Driver
GPU
3D X Server
Application
libGL VirtualGL Xlib
GLX
OpenGL Images
Driver
GPU, monitor attached
Network
X11
Events
X11
Cmds
Proxy X
Server Images
Client App
![Page 30: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/30.jpg)
EXAMPLE OF REMOTING SOLUTIONS TurboVNC + VirtualGL
+ Open source solution
+ Compressed image transport
+ Remote GPU accelerates OpenGL
http://www.virtualgl.org
NICE DCV
+ Commercial grade product
+ H264 encoded video stream
http://www.nice-software.com/products/dcv
![Page 31: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/31.jpg)
PARALLEL VISUALIZATION
![Page 32: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/32.jpg)
PARALLEL VISUALIZATION Domain decomposition
Parallelism at multiple levels
Filtering
Rendering
- Both supported by VisIt & Paraview
- Heavy lifting already done!
- Typically biggest challenge: Setup in parallel environment
- Both tools provide support for most common cases
- Both VisIt & Paraview MPI parallel -> need custom build
![Page 33: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/33.jpg)
PARALLEL COMPOSITING WITH ICET
• Each node renders fraction of image
• Sort last compositing
• Highly scalable
• Widely used (Paraview, VisIt .. )
http://icet.sandia.gov
![Page 34: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/34.jpg)
IN-SITU VISUALIZATION
![Page 35: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/35.jpg)
LEGACY WORKFLOW
Sim Sim Sim
Viz Viz Viz
Time
File system pressure
Long time-to-science
![Page 36: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/36.jpg)
PIPELINED IN-SITU ANALYSIS
Sim Sim Sim
Viz Viz Viz
Time
Reduced file system consumption
Early detection of faulty runs
Reduced
![Page 37: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/37.jpg)
DIFFERENT VISUALIZATION SCENARIOS
1) Legacy workflow
Separate compute & viz system
Communication via filesystem
2) Partitioned HPC system
Different nodes for viz & compute
Communication via High Perf Network
3) Co-Processing
Compute on GPU, viz on CPU
Compute and visualization on same GPU
HPC System
Compute GPU
Viz System
Viz GPU Filesystem
HPC System
Compute GPU
Viz GPU
Network
HPC System
Compute+Viz
nodes
![Page 38: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/38.jpg)
SUPPORT FOR IN-SITU VISUALIZATION Paraview: Catalyst
VisIt: LibSim
Instrument application to expose data
Paraview: Adaptor
LibSim: Data access callbacks
Potential for interactive steering
S5815, today, 3:30: Programming Pointers to Optimize your In-Situ Visualization Pipeline
S5710, Fri 9:30: In-Situ Data Analysis and Visualization: ParaView, Calalyst and VTK-m
![Page 39: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/39.jpg)
SUMMARY Visualization is more than rendering
Filtering often expensive component
Fast rendering can enable new applications (interactive supercomputing)
Tesla systems can be used for rendering On some systems requires GOM=All ON
Xserver or EGL
Remote visualization Supported by common tools
Take advantage of available hardware (H264 encoder)
Parallel Rendering Use IceT for compositing
In-Situ Visualization Supported by common tools
![Page 40: SCIENTIFIC VISUALIZATION ON GPU CLUSTERS](https://reader031.vdocument.in/reader031/viewer/2022012418/61735058b065b928a712f4e7/html5/thumbnails/40.jpg)
THANK YOU