Revolution

Enabling Large-Scale Collaborative Science

Outline

Introduction Visualization Applications Distributed, Parallel, Grid-based, and Collaborative

Visualization Collaborative Scientific Visualization Environments (CSVE) Future Directions

Revolution In Science

Pre-Internet• Theorize &/or experiment, alone or in small teams; publish paper.

Post-Internet• Construct and mine large databases of observational or simulation data.• Develop simulations, analyses, & synthesis.• Access specialized devices remotely.• Exchange information within multidisciplinary teams.

Image from CERN

Image from www.aip.org

Why Visualization?

Visualization is now seen as an integral part of modern computing

High performance computing generates vast quantities of data

High resolution measurement technologies generate vast quantities of data

Information systems incorporate large data sets and complex relations

We simply must harness our visual systems to aid us in understanding our data

What Is Visualization?

Medical Applications

From MRI, CT, Confocal Microscopes, …

We can visualize human anatomy at various scales

Curved Surface through the aorta tree. Visible Human Server, from R.D. Hersch at Ecole Polytechnique Fédérale de Lausanne

Optical nerve in the retina. Imaris software from B. Ehinger, Department of Ophthalmology, Lund University Hospital Torn ACL. Anonymous image.

Climate Applications

From simulators, satellites, measurement stations, …

We can visualize events, climate, and current weather

Satellite and surface image for January 19, 2004. Image from Unisys Weather.

These images show a comparison between two large El Niño events. The first begins in Oct '81 and the second in Oct '96. Image from NCAR.

Top-south view of 3-D volume of the simulated Andrew's radar reflectivity. Image from Y. Liu, McGill University.

Oil and Gas Applications

From simulators, seismic data sets, field measurements, …

We can visualize production, management, and exploration

Streamlines emanating from a virtual well show a three-dimensional oil flux. Image from Lawrence Berkeley National Laboratory.

Immersive visualization of horizons, faults, wells, and salt dome. Image from BP Visualization Center, University of Colorado.

Real-time cross-section planes where opacity is reduced in order to show values of interest. Image from HueSpace of Norway.

Molecular Applications

From simulators, experiments, measurements, …

We can visualize molecules, simulated values, and statistical measurements

The image depicts the electrostatic potential at each point of the Van der Waal's dot surface around aspirin. Image from Roger Sayle

Main chain hydrogen bonds and peptide bonds deviating more than some degree from planarityImage from Dirk Walther, UCSF.

Fancy CPK model. Atoms are made of various metals (C: gold, H: chrome, N: bronze, O: silver, S: brass). The ellipsoid (made of red glass) is the one with the smallest volume containing 70% of all atoms. The molecule is Trypsin Inhibitor. Image from L. Chiche .

Environmental Applications

From observations, experiments, measurements, …

We can visualize terrain, database information, and measurements

Populations of trees using a range of rendering techniques. Image from USDA Forest Service, Pacific Northwest Research Station .

Monitoring wind profile in Monterey Bay. Image from A. Pang, UCSC.

Patterns of recent forest management activities in the Northwest. Image from J. S. Nighbert, Oregon BLM.

Scientific Visualization

1987 NSF Report B.H. McCormick, T.A. DeFanti, and M.D. Brown, "Visualization in Scientific Computing," in Computer Graphics, Vol. 21, No. 6, (special issue).• Turning “firehoses” of data into a

visual representation• Enabling the scientist to “see the

unseen” Argued that investment in high

performance computing in US was wasted unless there was corresponding investment in visualization

Led to the development of several visualization software systems

One of the many visualization software systems created during this time. Developed by B. Hibbard. vis5d.sourceforge.net

Dataflow Visualization

Visualization represented as a pipeline:• Read data• Filter data• Map data• Render data• Display data

System realized in at least two ways:• Modular Visualization Environment• Toolkits or Libraries

Modular Visualization Environment

Modular Visualization Environments• IRIS Explorer, OpenDX, AVS, …• Visual programming paradigm -

allows easy experimentation which is what one needs in visualization

• Extensible – add your own modules

• Scientist uses ‘visual programming’ to connect modules together

AVS5 – www.avs.com

OpenDX – www.opendx.org

IRIS Explorer – www.nag.com

Visualization Libraries and Toolkits

Visualization libraries and toolkits OpenGL, Java3D, VTK, OpenRM,

Java3D, …• Provides the application programmer

an API• Scientist uses applications or

incorporates visualization code in own software

• Open source OpenGL

• Industry standard• Hardware acceleration• Basis for VTK, OpenRM, Java3D

Java3D• A mapping of OpenGL

OpenRM• Direct volume rendering

VTK• Bindings to Tcl, Python, Java

Virtual creatures from Stanford University using VTK – www.vtk.org

Visapult from LBNL usingOpenRM – www.openrm.org

vmd from NCSA using OpenGL – www.opengl.org

Cave using Java3D from the University of Calgary Java3D – java.sun.com/products/java-media/3D/

Visualization and Simulation

Visualization is a key tool in understanding the results of numerical simulations of complex phenomena

Use cases of visualization for simulation

• Pre-processing• Treat dataflow visualization

environment and simulation as separate activities

• Tracking• Replace data in visualization

pipeline with the simulation• Track behavior

• Steering• Include control module in

visualization pipeline• Simulation responds to

visualization environment• Post-processing

• Again, treat visualization and simulation as separate activities

Pre-process

Track

Steer

Post-process

Reservoir simulation using VTK from Geocap

Visualization and Observation

Visualization is a key tool in understanding observational data

Use cases of visualization for observational data• Monitor

• Monitor incoming observations

• Post-processing

• Treat visualization and observations as separate activities

• Integration

• Accept multiple input streams

Monitor

Integrate

Post-process

Monteray Bay monitoring from REINAS, UCSC

Distributed Visualization

Distributed visualization• Offload some computationally

intensive tasks• Couple the simulation with the

visualization• Typically, a single processor

is not powerful enough to run both the simulation and visualization

• Control and, in most cases, rendering will remain local

Types:• Single-processor• Multi-processor• Networked processors

These types can be used in combination

Visualization pipeline can be distributed in a number of ways

Loosely-coupled

Multi-processor - Parallel

Single-processor, possibly multi-processor

Issues

Multi-processor issues• Load balance• Latency• Decomposition, …

Control• Launching remote parts• Interacting with remote parts (steering problem)• Authorization• Authentication• Resource discovery

Data• Format

• Proprietary• Open Standards

• Compression• General purpose• Special purpose

Visual Network Computing/VizserverTM

Multi-processor – loosely coupled Access to SGI high performance

computing/graphics over network• Renders on remote devices• Remote framebuffer compressed

and distributed via TCP/IP over network

• Control over compression Features

• Application transparent• Shared-control• Platform/independent• Advanced visualization

environments• Scalable

Grid

Grid – Development and promotion of standard protocols to enable interoperability and shared infrastructure

• Globus toolkitTM – Open source reference implementation for building grid infrastructure and applications

• Global Grid Forum – Development of standard protocols and APIs for Grid computing

Layered Architecture• Collective – Managing multiple

resources to provide a ubiquitous infrastructure and services

• Resource – Sharing single resources, negotiating access, controlling use

• Connectivity – Talking to things securely

• Fabric – Controlling access and resources locally

Real-time visualization of advanced photon source data, Image from Argonne National Laboratory

Grid Service

Idea: A service with well-defined interface advertises itself in a distributed directory service• Application queries directory

service on how to interact with the service

Web Service• URI• Discovered by XML artifacts• Interactions through XML-based

messages• Standards WSDL, SOAP, …

Grid Service• Extends Web services• Standards – OSGA, OSGI

Grid Visualization

Use Grid Services to discover• Grid Visualization Service• Simulation Running on Grid• Data Stores on Grid

Grid Middleware• Compression• Native / XML Data

Grid Visualization Service• Simulation can register parameters

and data with the service• Data stores or databases can be

registered with the the service• Supports multiple clients• Service manages connections from

external clients• External clients can connect and

interact with data streams• Synchronizes connected clients

Parallel Visualization

Chromium• Open Source• Enables parallel rendering• Replaces systems OpenGL

driver• Industry standard API• Supports existing

applications• Streams of API

• Alters/Discards/Injects• Routes commands• Geometry is moved across

network• Rendered remotely

Visapult - LBNL• Parallel Volume Rendering• Uses OpenRM an industry

standard

Visapult, Image from LBNL

Chromium was created by Greg Humphreys, chromium.sourceforge.net

(a)

(b)

Collaborative Visualization

Radical collocation has proved highly successful• Manhattan Project• Space missions• Software development

Productivity Doubled!• Teasley et al, Michigan

But it requires:• Social disruption• Advance planning• …

Goal of Computer Supported Cooperative Work (CSCW):• Gain in productivity, but reduce

collocation requirement using electronic collaboration

Need to move away from seeing collaborative visualization as a group crowded around a display screen

Towards collaboration over network

CSCW Model

CSCW Model associates applications with approaches

Based on:• When?• Where?

Visualization• Real Time

• Same Place

– AVS, Amira, …

• Different Place

– What do we share?

» Display

» Visualization

» Process

– How many users/location?

Sharing Screen

Simple model• Broadcast display of application

to a set of passive users• Number of available technologies

• IRIS Explorer, AVS, …

• VNC – Virtual Network Computing

– RealVNC – www.realvnc.com

– tightVNC – www.tightvnc.com

VNC, from AT&T

Sharing Visualization

Share the visualization• Geometry is exchanged• Master/Slaves• Number of available applications

• COVISE, IRIS Explorer Advantages

• Greater involvement of collaborators

• Shared Control – Token Passing Disadvantages

• Can’t determine what collaborators are doing

• Limited collaboration

COVISE, from Dr. Ulrich Lang Computing Center University of Stuttgart Visualisation Department

Sharing Process

Each collaborator may participate in producing the visualization

Two variations:• Replicated

• Initial data sharing

• Parameters are interlinked

• Small network traffic

• Application tailored to individuals expertise

• CSVE• Interlinked

• Separate pipelines

• Cross wiring pipelines enables collaboration

• Greater flexibilty

• Varying network traffic

• COVISA

Issues

Portable• Different OS• Different Libraries/Toolkits/MVEs

Functionality• Data• Parameters• Algorithms• Applications

Participation• Joining/Leaving• Floor control• Privacy• WYSIWYTIS• Authentication

System• Performance• Scaling• Reliability• Robust• Security

CSVE, from Patrick O’Leary

COVISA, from Jason Wood, Visualization Scientist, University of Leeds

Access Grid

The Access Grid™ • Ensemble of resources

• Multimedia large-format displays,

• Presentation and interactive environments,

• Interfaces to Grid middleware and to visualization environments

VRVS• Desktop Web-based alternative

Advantages• Greater sense of involvement• Lower geek threshold• Used in combination with VNC

Access Grid, Image from www.accessgrid.org

VRVS, www.vrvs.org

CSVE

Collaborative Scientific Visualization Environment (CSVE)

• Facilitate Scientist - Computer Scientist or Small Group Interaction

• Open Source• Java• JMF• VTK

• Features• Interactive 3D Visualization• Streaming Audio/Video• Streaming Media• Desktop Capture• Chat• Whiteboard• Telepointer*• Remote Control Client*• Data Management*

… a research area expert

A visualization expert interacts with …

CSVE

Anastasia Mironova – Vis 2003 Interactive 3D Visualization

• Handles several data formats• Create/Manage isosurfaces,

slices, …• Simple tools for interacting with

visualization• Seamless network propagation of

visualization parameters

Manage visualization objects …

Create visualization objects …

CSVE

Brian Mullen – Vis 2003 Streaming Media

• Stream any mpeg, avi, mov file to collaborators

Streaming Audio/Video• Stream audio/video from two to

… collaborators Desktop Capture

• WYSIWIS not WYSIWYTIS

Stream scientific videos

Stream audio/video to collaborators

See what they are looking at

CSVE

Scientific Database Currently:

• Relational Database – MySQL, Oracle, …

• Flat files Moving to Meta Catalogue

• Based on an extension of XML Why XML?

• Accepted way of describing things for the Web and the Grid.

• Good at describing things because• Wide range of concepts can be

captured in this way.• It provides a basis for validators,

transformers, parsers, analyzers, displayers, …

• So simple• This is why HTML became so

widely used.• Can teach anyone to use it in a

short period of time.

<?xml version='1.0'?> <list> <recipe> <recipe_name>Chocolate Chip Bars</recipe_name> <author>Carol Schmidt</author> <meal>Dinner <course>Dessert</course> </meal> <ingredients> <item>2/3 C butter</item> <item>2 C brown sugar</item> <item>1 tsp vanilla</item> <item>1 3/4 C unsifted all-purpose flour</item> <item>1 1/2 tsp baking powder</item> <item>1/2 tsp salt</item> <item>3 eggs</item> <item>1/2 C chopped nuts</item> <item>2 cups (12-oz pkg.) semi-sweet choc. chips</item> </ingredients> <directions> Preheat oven to 350 degrees. Melt butter; combine with brown sugar and vanilla in large mixing bowl. Set aside to cool. Combine flour, baking powder, and salt; set aside. Add eggs to cooled sugar mixture; beat well. Stir in reserved dry ingredients, nuts, and chips. Spread in greased 13-by-9-inch pan. Bake for 25 to 30 minutes until golden brown; cool. Cut into squares. </directions> </recipe></list>

CSVE

Message Passing• Objects through bit-stream• Same underlying principles as

remote object broker or RMI

• No parsing

• Flexible

• Extensible

• Efficient

• No parsing!• Moving to XML messages

• The way messages are passed by Grid- and Web-services

• Slower

• Standard format

• Requires parsing messages built into Java

3 Tier Architecture

Application Neuroscience

Pain• Quality of Life• Neurochemical Changes

Image Reconstruction• Removal of Noise and Artifacts• Deconvolution of Light Source• Segmentation of Data

Visualization Techniques:• Maximum Intensity Projection

(MIP)• Volume Visualization

Application Neuroscience

Application Cancer

Bone Cancer• Bone Destruction• Tumor Burdon

Image Reconstruction• Removal of Noise and Artifacts• Edge Detection• Automation• Segmentation of Data

Visualization Techniques:• Isosurfaces• Volume Visualization

CSVE

Interactive Visualization

Desktop CapturePortable•Windows•Apple•Linux

Streaming Media

Additional Applications

CSVE

CSVE

Future Work• Grid Protocol Based

• Resource discovery

– Databases

– Simulations

– Smart Instruments

– Visualization Resources

• Data exchange

• Message passing

• Server as a Grid-service• Remote Control• OpenRM – Direct Volume

Visualization Version• More Visualization Techniques• More sophisticated data

management

Acknowledgements

NSF MRI grant, 0215583, and a NSF REU Supplement to the grant NSF EPSCoR Alaska for funding both Anastasia Mironova’s and Brian

Mullen’s summer research internships The University of Alaska Anchorage (UAA) Office of Undergraduate

Research and Scholarship, Office of Research and Graduate Studies, and Dr. Hilary Davies, whom through Discovery Grants and travel funds made it possible for both Mironova and Mullen to present their work at Visualization 2003

Jonathan Snelling, supported by a NSF REU Supplement, for his work on a multi-document graphical interfaces

Brian Mullen for his development of streaming audio/video tools (he put the “C” in CSVE)

Anastasia Mironova for her development of volume visualization tools, integrating additional data formats, and winning Best Poster at Visualization 2003

My CS 401 Software Engineering class at UAA (Nicholas Armstrong-Crews, Jan Reitspies, Kevin Dickerson, William Sistar, John Vicente, Jeffrey Woods, Daniel Stokley, Justin Dieters, Christopher Johnson, Mark Blum, Shannon Smith, Brandon Douthit-Wood, Shane Ursani, Nathaniel Freeburg, and Christopher Ulmer).