visualization clusters using_commodity_hardware
Post on 10-May-2015
155 Views
Preview:
DESCRIPTION
TRANSCRIPT
0 1 / 1 2 / 2 0 1 0
This paper provides a high-level description of the various
types of visualization clusters and briefly covers the use of
commodity hardware to build cost effective solutions. It lists
a selection of commercial and open source applications by
field or specialty, with in-depth descriptions of some of
those applications with an emphasis on the unique value
provided by the cluster solution. While describing different
visualization clusters it focuses on hybrid computational
multi-display clusters. (VizWalls)
Visualization Clusters using Commodity Hardware
2 Visualization Clusters using Commodity Hardware
EXECUTIVE SUMMARY ........................................................................................................................................... 3
VISUALIZATION CLUSTERS ..................................................................................................................................... 4
RENDERING OPTIONS ...................................................................................................................................................... 4
Single System Rendering ........................................................................................................................................ 4
Parallel Rendering .................................................................................................................................................. 5
IMAGE DISPLAY OPTIONS ................................................................................................................................................. 5
Single Workstations ............................................................................................................................................... 5
Large Single Display ............................................................................................................................................... 5
Seamless Multi-Display Systems ............................................................................................................................ 6
Tiled Liquid Crystal Display Panel Display Wall ...................................................................................................... 6
COMBINED DISPLAY WALL AND RENDERING CLUSTER ............................................................................................................ 6
Typical Hardware ................................................................................................................................................... 7 Software .............................................................................................................................................................................. 9 Cluster Management System ............................................................................................................................................... 9 Visualization Middleware – Open Source ............................................................................................................................ 9
Commercial Visualization Applications ................................................................................................................ 11
Open Source Modeling ......................................................................................................................................... 11
COLLABORATION .......................................................................................................................................................... 12
Access Grid ........................................................................................................................................................... 12
OptIPortal ............................................................................................................................................................ 12
ABOUT X-ISS ........................................................................................................................................................ 13
3 Visualization Clusters using Commodity Hardware
Executive Summary
With the exponential growth of affordable compute power has come a similar growth in the ability to solve
more and more complex problems. These complex problems require us to visualize and interact with
increasingly large data set sizes. Dell is a leader in the high performance compute arena and is now
bringing leading edge solutions for users to be able to visualize these massive datasets. The paper
describes the various types of visualization solutions in general and provides specific details on high
performance visualization clusters and walls. It describes the various components of these ‘VizWalls’ and
how customers can leverage these new solutions.
4 Visualization Clusters using Commodity Hardware
Visualization Clusters
Visualization is an important process for making large data sets accessible to humans. It allows us to
explore, interact and understand complex relationships and attributes from the large volume of data
created by scientific instruments and high performance computer simulations and other applications.
Commodity High Performance Computing (HPC) environments have allowed many scientists to use
scientific visualization techniques that were once reserved for the few. Natural scientists may render
molecules; ecologists may model large-scale climate change; and investigators in the life sciences have
access to increasingly detailed datasets for deepening our knowledge and understanding. Creating and
manipulating highly dense and complex images is essential so humans can identify patterns, singularities,
and detail not apparent from the raw data. Visualization clusters have become an indispensable tool for
viewing this complex data.
Just a few years ago, setting up a HPC visualization cluster and display wall required either a large
budget or a lot of do-it-yourself effort. Today, vendors such as Dell provide turnkey visualization clusters
built with low-cost commodity hardware that can be up and running in a matter of days.
Rendering Options
The first aspect of visualization is how the image for display is actually generated (also called rendering).
This section describes the two common methods and technologies used for this process.
Single System Rendering
A common use case in High Performance Computing (HPC) Clusters is the visualization of
computationally intensive problems as static images or animations. This sort of application is
characterized by large input datasets and comparatively small output sizes, such as the visualization of
seismic data. The cluster nodes frequently have no specialized graphics hardware. Long running tasks on
many cluster nodes generate image files that are then displayed on separate display devices or systems.
See Figure 1 for an illustration. In this scenario, the display devices are workstations with good
performance, and they may use high performance graphics cards to display on one or two high-resolution
monitors. The compute and display power of this workstation is typically sufficient for the user to render
and interpret the images from the datasets interactively.
Hundreds or Thousands of
Nodes with Petabytes of Data Graphics Workstations
Local
Storage of
generated
images
Figure 1 - Diagram of Remote Visualization System
5 Visualization Clusters using Commodity Hardware
Parallel Rendering
Another common use case is parallel rendering using graphics hardware without attached displays on
HPCC compute nodes. This need usually arises if the complexity, size, and density of the image make it
difficult for one system to allow effective interaction with the user. In the case of a complex/dense image,
even though the final image may still be on the same display size as the previous example, the compute
and graphics power needed to create and/or interactively manipulate the image (rotate, shade, live
simulation etc.) is not available in a single graphics workstation.
For example, a large 3D model is divided up between available resources on the cluster and a single
workstation is used to display the rendered images. Each of the compute nodes is responsible for
rendering a portion of the image, with the resulting pieces assembled on a single workstation. This type of
parallel rendering known as data parallelism, often used in CAD and medical visualization applications,
provides the user with the required ability to manipulate large models in real time. See Figure 2 below.
A slightly different approach, known as temporal parallelism, divides the rendering into many sequential
frames with each node rendering a complete image. This technique is ideal for animation where real-time
rendering is not as important as total rendering time.
An excellent description of both of these approaches is available in the Dell Power Solutions article
Parallel rendering Technologies for HPC Clusters, available at
http://www.dell.com/downloads/global/power/ps4q07-20070550-Ou.pdf
HPCC cluster with specialized
graphics hardware renders
images
Sends Request (ie: rotate image)Distributes work to
compute nodes
Workload manager
Returns rendered tiles
Graphics workstation
Assembles Complete
Image
High Bandwidth Network
Figure 2 - Parallel Rendering using Data Parallelism
Image Display Options
Single Workstations
The most basic option for display of generated images is the single workstation with one or two high-
resolution screens. Lowest in cost and complexity of the solutions discussed here, this is most useful for
individual users and users collaborating with shared screens with VNC or NoMachine software.
Limited to 4-8 megapixels, these solutions force the user to choose between detailed views of small
datasets and overviews of large datasets. They are not well suited for in-person collaboration with more
than a couple of people.
Large Single Display
Moving up in scale, cost, and complexity, large single displays are available. These displays use LCD
projectors to display large images. Single projector displays are currently limited to about 8 megapixels,
6 Visualization Clusters using Commodity Hardware
nearly four times the resolution of HD video. These displays are useful for larger groups of people viewing
less detailed images from several feet distant, but sacrifice detailed viewing for size.
Seamless Multi-Display Systems
More costly and complex are multi-projection displays. Usually rear projected, these displays can achieve
both high resolution and large, seamless displays. These are the most complex and expensive of the
display types, require precise alignment and consume 300-400 watts per projector, making them the most
power hungry of the group and requiring the most cooling. At 7-9 feet deep, they also require the most
space.
Because of the multiple displays, these solutions require dedicated compute and display servers and
specialized software identical to the tiled LCD panel display wall below.
These displays are very well suited for in-person collaborative visualization and viewing from both
moderate distances and close-up.
Tiled Liquid Crystal Display Panel Display Wall
With cost and complexity between large single displays and seamless multi-display systems, tiled LCD
panel display walls are becoming a very popular solution.
As with multi-projector systems, tiled LCD panels can be built to arbitrary viewing sizes and resolutions. A
modest wall built with nine 2460x1600 30" monitors can contain over 38 megapixels with a 4.5 x 7 foot
viewing area. A 40-monitor wall can have over 168 megapixels in an 8’ x 18’ viewing area. A large wall
like Stallion at the Texas Advanced Computing Center at the University of Texas at Austin has over 307
megapixels.
At less than a foot deep, these walls are slim compared to the multi-projection displays. Power and
cooling is lower also, with each display consuming less than 200 watts.
Large, high-resolution displays like these enable the observer to visualize very detailed information with
clarity while allowing it to be visualized in the context of a large body of data.
Combined Display Wall and Rendering Cluster
A visualization cluster consists of a standard HPC cluster with the addition of high-resolution graphics
cards and monitors arranged in a grid. Typically, an additional graphics workstation is used to control the
wall. The cluster uses standard network interconnects (GbE, IB) and storage.
The addition of specialized visualization software makes the cluster useful for a large range of
commercial and academic applications including medical imaging, cartography, engineering simulations,
geologic exploration, and biologic modeling.
7 Visualization Clusters using Commodity Hardware
Typical Hardware
A visualization wall is typically driven by a standard HPC cluster using graphics workstations for compute
nodes installed with one or more high performance graphics cards. Storage for images and models can
be direct attached to the head node or remotely accessed from enterprise storage pools. For smaller
walls, as show in figure 3, gigabit Ethernet is sufficient for both the management network and the cluster
data fabric. The wall control workstation may also be used as the cluster head node. Ethernet
connection to the control workstation should be at least dual bonded GbE or 10GbE.
Figure 3- Small VizWall Example
For larger walls, with 8 more display nodes such as the one shown in Figure 4, a separate high-speed
network should be added for the cluster fabric.
Figure 4- Large VizWall Example
8 Visualization Clusters using Commodity Hardware
At the extremely large end of the
scale is Stallion at the Texas
Advanced Computing Center.
Stallion has a resolution of over
307 megapixels, provided by 75
30” Dell LCD displays, each with
2650 x 1600 pixels.
The wall is driven by 23 Dell XPS
workstations connected with an
Inifiniband network.
Stallion, build with all commodity
components processes datasets
of a massive scale with interactive
visualization.
At the more modest end of the scale is the 3x3 wall at the left. This wall, also build with Dell 30” high resolution monitors is driven by 5 Dell R5400 rack mounted graphics workstations. Because of the smaller size of the wall, GbE is used for a cluster interconnect, although two ports are bonded on the display controller node. Walls of this size and larger are now
available from Dell as compete turnkey
systems including monitor mounts,
software and installation.
Photo courtesy of: Texas Advanced Computing Center (TACC)
Photo courtesy of: X-ISS, Houston, Texas
9 Visualization Clusters using Commodity Hardware
Software
There are four major software components necessary for a visualization cluster: an operating system, a
cluster management system, visualization middleware and application software.
Cluster Management System
A Cluster Management System (CMS) is used as in a traditional HPC cluster to reduce the complexity of
system provisioning and management. Windows HPC Server 2008 is the CMS of choice for Windows
clusters, but still enjoys less commercial support for viz walls.
For a Linux viz wall, the clear choice for a CMS is Rocks, open source software developed at the
University of California, San Diego. More tiled display walls are constructed using Rocks and its support
for building tiled display walls through its viz roll than any other solution.
Dell provides Linux viz walls using Rocks+GPU from Clustercorp which combines the viz roll and the
CUDA roll with support for GPGPU clusters. The full commercial support from Clustercorp and
professional installation from Dell contribute
to a successful deployment.
Visualization Middleware – Open Source
Displaying across multiple monitors driven by
multiple nodes requires specialized software.
SAGE
SAGE refers to the Scalable Adaptive
Graphics Environment from the Electronic
Visualization Laboratory (EVL) at the
University of Illinois at Chicago.
SAGE is a graphics streaming architecture for
supporting collaborative scientific visualization
environments with potentially hundreds of
megapixels of contiguous display resolution. In
collaborative scientific visualization, it is crucial
to share high-resolution imagery as well as
high-definition video among groups of collaborators at
local or remote sites. (Scalable Adaptive Graphics
Environment -EVL website).
Unlike many display drivers that display the output of a single application on an entire wall, SAGE can
allow many different applications to display on different areas of a high-resolution display wall
simultaneously. It allows simultaneous display of pixel streams from various sources to be tiled across a
wall making it an excellent choice for collaboration.
Linux Software Stack
Application Software
Visualization Tools SAGE, Chromium, DMX
Cluster Management System
Cluster Corp Rocks+
Operating System Red Hat or CentOS
Windows Software Stack
Application Software
Visualization Tools SAGE, Chromium
Cluster Management System
HPC08
Operating System Windows Server 2008
Table 1 - Typical Software Stacks
10 Visualization Clusters using Commodity Hardware
Many applications support SAGE, including images and movie viewers, scientific visualization, and
collaboration. For applications without built in support, SAGE comes bundled with OpenGL capture, a
package that enables software that uses OpenGL to display through SAGE without source code
modifications.
SAGE includes a graphical user interface that models the tiled display, allowing users to position
application windows. Collaborators may each run a user interface and interact with the display at the
same time.
SAGE is under active development and is used by dozens of visualization applications. It is available for
Linux, Windows, and Mac OS X. It was recently used by a team from LSU as part of an application that
won first prize at the IEEE Scalable Computing Challenge in Shanghai.
Chromium
Chromium is a flexible framework for scalable real-time rendering on clusters of workstations, derived
from the Stanford WireGL project code base. It intercepts OpenGL calls and automatically divides the
work up between cluster nodes, each with its own display(s).
It is a completely extensible architecture, so that parallel rendering algorithms can be implemented on
clusters with ease. (Chromium Documentation) It runs on Windows, Linux, and several Unix distributions.
First released in 2001, the current version, Chromium 1.9, was released in 2006.
DMX
Ximerama is an X windows extension that presents multiple displays as one contiguous screen. The size
of the aggregated screen is limited by the number of physical devices that are supported in a single
machine.
DMX is a proxy X server that extends multi-head support for displays attached to different machines.
When used with Ximerama the displays are presented to the user as a single screen. DMX has been
integrated into the X.org server software.
ParaView
ParaView is an open-source, multi-platform data analysis and visualization application. ParaView users
can quickly build visualizations to analyze their data using qualitative and quantitative techniques. The
data exploration can be done interactively in 3D or programmatically using ParaView's batch processing
capabilities.
ParaView was developed to analyze extremely large datasets using distributed memory computing
resources. It can be run on supercomputers to analyze datasets of terascale as well as on laptops for
smaller data. (Paraview - Open-source Scientific Visualization)
ParaView is currently maintained and is available for Linux and Windows.
VR Juggler
VR Juggler can be used on complex multi-screen systems running on clusters, high-end workstations,
and supercomputers. VR Juggler allows applications to execute in many VR system configurations,
including desktop VR, HMD, CAVE™-like devices, and Powerwall™-like devices. VR Juggler 2.0
supports IRIX, Linux, Windows, FreeBSD, Solaris, and Mac OS X. (The VR Juggler Suite website)
11 Visualization Clusters using Commodity Hardware
Commercial Visualization Applications
Wolfram Research’s gridMathematica
gridMathematica adds extra computation kernels and automated network-distribution tools to Wolfram
Research’s Mathematica. gridMathematica runs tasks in parallel, over multiple CPUs with fully
automated process coordination and management and no need for code changes. Mathematica supports
multiple Windows, Macintosh, Linux, and Solaris platforms.
gridMathematica Server provides 64-bit optimizations, including integration with many cluster
management systems such as Altair PBS Professional, Microsoft Windows Compute Cluster Server,
Microsoft HPC Server, Platform LSF, and Sun Grid Engine. gridMathematica Server also provides
MathLink's support for both Gigabit and high-speed data networks. (Wolfram gridMathematica website)
CEI’s EnSight
EnSight provides an icon-based interface that allows the user to learn the program quickly and move
easily into layers of increased functionality. EnSight works on all major computing platforms and supports
interfaces to most major CAE programs and data formats.
EnSight can be run standalone or distributed using client-server operation on Windows, Apple, Linux,
SGI, SUN, IBM, and HP platforms. It runs in parallel on up to two processors on shared-memory
computer systems.
EnSight reads multiple data sets and enables the user to run comparisons between them. The data can
originate from different solvers and/or different disciplines. EnSight post-processes data remotely with
client-server operation. (CEI EnSight website)
VCOVISE
COVISE is a modular visualization software program that supports Virtual Reality and collaborative
networking capabilities. It is a system platform for constant use of Virtual and Augmented Reality
technologies.
The modular structure for add-ons to the basic COVISE module ensures successful customization for
suite-specific needs. Modules are available for fluid dynamics and structural mechanics readers, virtual
reality rendering, collaborative engineering, volume rendering, web browser view rendering, batch
processing, viewing VRML files, development environment, casting and molding simulation, multi-body
systems, QA and tolerance analysis, and CAD file visualization.
Open Source Modeling
VMD
VMD is designed for modeling, visualization, and analysis of biological systems such as proteins, nucleic
acids, lipid bilayer assemblies, etc. It may be used to view more general molecules, as VMD can read
standard Protein Data Bank (PDB) files and display the contained structure. VMD provides a wide variety
of methods for rendering and coloring a molecule: simple points and lines, CPK spheres and cylinders,
licorice bonds, backbone tubes and ribbons, cartoon drawings, and others. VMD can be used to animate
and analyze the trajectory of a molecular dynamics (MD) simulation. In particular, VMD can act as a
graphical front end for an external MD program by displaying and animating a molecule undergoing
simulation on a remote computer. http://www.ks.uiuc.edu/Research/vmd/
12 Visualization Clusters using Commodity Hardware
Collaboration
One of the most exciting uses of a viz wall is for collaboration between groups of users. Each can display
graphics on the screen simultaneously to illustrate differing views of the data, compare the results of
different calculations, or create synergistic displays of data from different disciplines. This is even more
exciting when the participants are geographically dispersed, using part of the wall for teleconferencing
while using other areas on the wall for graphics and demonstrations.
Two of the most popular collaboration environments are Access Grid and OptlPortal, both open source.
Access Grid
The Access Grid® includes multimedia large-format displays, presentation and interactive environments,
and interfaces to Grid middleware and visualization environments. These resources are used to support
group-to-group interactions across the Grid.
AG3 is the newest version of the Access Grid software. It has been updated to conform to standard
Internet technologies and protocols and to maximize robustness, performance, and interoperability. All
network connections are encrypted by default using an X509 certificate at the venue server. Users do not
require a certificate. Access Grid is supported on Windows, Apple OSX, and a number of Linux
platforms. (Access Grid website)
OptIPortal
An OptIPortal is a visualization cluster that can be deployed on a variety of hardware platforms.
Functionally, one or more OptIPortals can be used to view high definition static images, video, or in
streaming mode in both 2D and 3D environments. 3D OptIPortals in use currently use COVISE and
CGLX as middleware. 2D OptIPortals in use currently run CGLX or SAGE.
Actions on the OptIPortals can be concurrent or persistent. Clusters of OptIPortal computers (nodes) can
run general computations as well as display and collaboration management. OptIPortal supports SAGE,
DMX, Chromium, CGLX, and OpenCOVER (Covise) as middleware. (OptIPortal website)
13 Visualization Clusters using Commodity Hardware
About X-ISS
eXcellence in IS Solutions, or X-ISS, is a professional services firm located in Houston, Texas, that
specializes in delivering high-performance computing solutions. X-ISS expertise includes designing,
deploying, managing, and benchmarking clusters of all types and sizes, including various interconnects,
cluster management suites, distributed resource managers, and popular HPC applications.
X-ISS manages and designs HPC Linux and Windows Compute Cluster Server based-clusters for oil and
gas, government, education, life sciences, and numerous other vertical markets. X-ISS is vendor-agnostic
and works with any hardware vendor or independent software vendor (ISV) to deliver cluster solutions.
X-ISS can help lower total cost of ownership, reduce administrative costs, optimize performance of both
hardware and applications, provide independent benchmarks of ISV applications, and manage HPC
clusters from small work group clusters through 12,000 core high throughput compute facilities.
If you have any questions regarding this white paper, please contact hpc@x-iss.com.
top related