department of computing science shuttle radar topography
TRANSCRIPT
1
Multi-modal Exploration of
Large Scientific Data Using
Virtual Reality
Dr. Pierre Boulanger
Department of Computing Science
University of Alberta
Department of Computing Science
University of Alberta
Shuttle Radar Topography Mission
(SRTM)
February 11, 2000, the Shuttle Radar
Topography Mission (SRTM) was
launched into space as part of one of
the payload of the Shuttle Endeavor.
Using a new radar sweeping technique
most of the Earth's surfaces was
digitized in 3D in approximately 10
days.
SRTM acquired enough data during its
mission to obtain a near-global high-
resolution database of the Earth's
topography.
Terrain Model of Mount St. Helens
Terrain Model Rendering
Terrain Model After Compression
and Hole Filling
Low Altitude Airflow Over Mount St.
Helens
Aburra Valley Colombia
CFD Model
Convective Winds Simulations
Based on Landsat IR Data
2
Virtual Analysis of a Francis Turbine
at “La Herradura” in Colombia• The main objective of the DIFRANCI Project is to apply a condition
assessment methodology following holistic approach to the
maintenance of the Francis turbines of "La Herradura" hydropower
plant in Colombia.
• Project in collaboration between:• Empresas Públicas de Medellín
• Colombian Agency for Science and Technology
• EAFIT University, Medellin, Colombia
• EPFL, Lausanne, Swizerland
• UofA, Edmonton, Canada
Digitizing the Turbine Using a Hand Held
Scanner
Scanning Using Handy Scan from
Creaform 3D
Final Scanned Reversed Engineered
Model
Scanned Model of the Turbine
Extracted FEM Mesh
Rapid Virtual Prototyping
• Once a 3D model is created, virtual
prototyping allows product testing without
the need to build a real prototype
• Allows for shape and functional
optimization
• Allows to tract the complete life cycle of a
product
• Rapid Virtual Prototyping requires powerful
computing infrastructure especially if it is
interactive
CFD Simulation of Francis Turbine
ProjectParticle Flow
3
Pressure Variations vs Time Results of CFD Analysis
Wall Pressure Comparisons of the computed pressure
recovery coefficient with the experimental
values, medium size mesh, 1.15ψ= in the
case of the FLINDT draft tube.
The UofA/EAFIT Virtual Wind Tunnel Definition of Interactive CFD
Need: A set of tools that allow the
designer to test interactively the
behavior of a design under various flow
conditions.
Task: The main task performed by an
interactive CFD system can be defined
as solving simulations of fluid flow for an
object, given the possible scenarios
defined by the user and to display the
results interactively.
User/Consumer: The target market is
none other than designers interested in
testing their design’s behavior under
fluid flow conditions to optimize and test
their design.
Interactive CFD User needsUofA/EAFIT Virtual Wind Tunnel
Architecture
4
Standard Scientific
Visualization/Simulation Pipeline
Data ImageFilter Map Render
Simulation
Parameters
Data
Repository
High Speed Network
Simulator
Remote
Desktop
Problems With Current Practice
• Hard and expensive process to determine
dominant parameters in a CFD simulation
model.
• Simulation runs cannot be steered
resulting in useless computations.
• It is like working blind.
• Current pipeline do not allow
collaboration. It is always after the fact
that the simulation data is analyzed and
shared by a group of engineer.
Even Better: Collaborative
Visualization/Simulation Steering Environment
Simulation
Parameters
Filter Map Render
User1
ImageData
Client 1
Simulation
Server
Filter Map Render
User n
ImageData
Client n
UofA Advanced Collaborative
Immersive Environments
3D Sound
Rendering
Input
Sensors
3D
Graphic
Rendering
Massive
Storage
Haptic
Rendering
New VizRoom
AMMI Lab Local
High Speed Network
Definition of Virtual Reality
• A virtual reality system is an interface
between a human and a machine capable
of creating a real-time sensory experience
of real and artificial worlds through the
various human sensory channels.
• These sensory channels for man are:
Vision, Audition, Touch, Smell, and Taste.
Burdea, 1993
Definition of Real-Time
Real-time, in virtual reality, means
that the computer system can detect
the input of the user and react to it
fast enough so that it appears to be
instantaneous.
5
How is VR Different than CG?
• Objects in the environment have a
strong sense of spatial presence,
creating the effect that the objects
exist independently of the user.
• Control of interaction within the
environment is often through direct
manipulation of objects as in the real
world.
How is VR Different than CG?
• The computer interface is “hidden” in
the sense that the user interacts with
objects in the environment rather
than a computer which controls
objects in the environment.
• The user is immersed in the
environment, i.e., the user
experiences the environment from
within.
Immersion=“Presence”
• Presence is a state of
consciousness where the
human actor has a sense of
being in the location
specified by the displays.
• The unique feature of
"virtual reality" systems is
that they are general
purpose presence
transforming machines.
Multi-modal systems• use more than one sense or mode of interaction
• e.g. visual and aural senses: a text processor may speak
the words as well as echoing them to the screen
Multi-media systems• use more than one media to communicate information
• e.g. a computer-based teaching system
- may use video, animation, text and still images
- different media all using visual mode of interaction
- may also use sounds, both speech and non-speech
Multimodal vs. Multimedia
Multimodal Human–Computer
Interaction
Information processing
Information processing
Human
Computer
Internal perception / action feedback loop
AI agents internal decision loop
Multimedia output
Multimodal inputInterface
Level of Interactivity
• Reactive: where little user control
takes place over the content's
structure with program directed
options and feedback
• Co-active: providing user control for
sequence, timings and style.
• Pro-active: where the user controls
both structure and content at most of
the levels.
6
First Level of Function Segregation Virtual Wind Tunnel Interaction Map
n-DData
VirtualWorld
Mapping
Data Exploration and the Mapping
Problem
Visual
Haptic
Sound
Visualization
Westgrid 4K x 2K Display
Visualization Toolkits
• Computation/Analysis + Visualization
• NIH Image and its PC version Scion
• Matlab
• Programming Toolkits
• The Visualization ToolKit (VTK)
• Insight ToolKit (ITK)
• VisAD and Vis5D (also “visualization spreadsheet”)
• SCIRun
• Graphical Programming Toolkits
• Open Data Explorer (OpenDX)
• Paraview
• Advanced Visual Systems (AVS/Express)
• Amira
• Slicer
Multivariate Display Techniques
• Glyphs: 2D Scalar
• Heterogeneous Techniques: 2D and 3D
• Texture
• Spot Noise (van Wijk), Healey, Ware
• Layering
• 2D Scalar: Slivers, DDS (3D?)
• 2D Scalar, Vector, Tensor: Laidlaw, Crawfis
• Problem Reduction
• Dimensional reduction, Cluster analysis
• Smart Particles
• Time and space multiplexing
• Mapping different fields over time
• Magic
7
Glyph: Flow Probe Multiple Views in Space
Definition of Haptic Gibson (1966)
• A haptic system is defined as "The
sensibility of the individual to the world
adjacent to his body by use of his
body".
• The haptic perceptual system is
unusual in that it can include the
sensory receptors from the whole body
and is closely linked to the movement
of the body so can have a direct effect
on the world being perceived.
Human Haptics
• Two complementary channels:
~ Tactile
• Strictly responsible for the variation
of the cutaneous stimuli
• Presents spatial distribution of
forces
~ Kinesthetic (Proprioception)
• Refers to the human perception of
one’s own body position and motion
• Presents only the net force
information
Tactile Display
• Skin sensation is essential for many manipulation and exploration tasks, for example, medical palpation. Tactile display devices stimulate the skin to generate these sensations of contact.
• The skin responds to several distributed physical quantities:
1. High-frequency vibrations: Surface texture, slip, impact, and puncture.
2. Small-scale shape or pressure distribution
3. Thermal properties
CyberTouch of Immersion
• CyberTouch™ is a tactile feedback option for Immersion's CyberGlove.
• It features small vibrotactile stimulators on each finger and the palm of the CyberGlove.
• Each stimulator can be individually programmed to vary the strength of touch sensation.
• The array of stimulators can generate simple sensations such as pulses or sustained vibration, and they can be used in combination to produce complex tactile feedback patterns.
8
Other Tactile Displays
• Thermal properties: We infer
material composition and
temperature difference. Thermal
display devices usually based
on Peltier thermoelectric
coolers.
• Many other tactile display
modalities: electro-rheological
devices for conveying
compliance, electro-cutaneous
stimulators, ultrasonic friction
displays, and rotating disks for
creating slip sensations.
Haptic Rendering of Surfaces
The Models of the Probe
a line segment
a point
a 3D object
State of the Art
• Difficult to simulate proprioception on the
entire body
Fingertip
Arm +
2 fingers
Wrist
Foot
VR Architecture of CoRSAIRe/CFD
Environment at LIMSI
Menelas 2010
CFD Exploration Using LIMSI
CoRSAIRe System
9
CFD Exploration Using CoRSAIRe
System Haptic Data Rendering-I
Tourque Nulling Transverse Damping
Pao and Lawrence, 1998
Haptic Data Rendering-II
Relative Drag Feature Shift
Pao and Lawrence, 1998 van Reimersdahl et al., 2003
AMMI Lab Multi-Modal Interface
for CFD (Visual and Sound)
Type of Modalities
Modalities used for the interface
• Visual Mono/Stereo/CAVE
• Haptic
• Perception of fluids flow
• Objects manipulations
• Setting of boundary conditions
• Sonification of fluids
Multi-Modal Exploration of CFD
Flow
10
Project Background
Input:
• Fluid field with velocity vector, pressure, and other data
• Changes with time
Output:
• Sound characterizing the given fluid field
• Ambient: global to the whole field
• Local: at the point or area of interaction
• Local region: particles of the specific subset area around the pointer contribute to the sound
Multi-Modal Rendering System
Structure
Solution Data Server
Max/MSP Program
Main Program as
Max/MSP object
Visualization
Program
Haptic Program
Haptic DeviceImage
Sound
Haptic Rendering
Read from the haptic device and sends pointer info to the sound and visual programs
• Pointer position and orientation (converted to the data field dimensions)
• Buttons: interaction sphere diameter- local region
Render a force feedback:
• Virtual walls: provides a force disallowing movement of the device outside of the data field boundary
• Other feedback possible: produce a force that is proportional to the flow density and its direction
Visual Rendering
• Displays vector field, virtual pointer
(microphone) and interaction sphere
• SGI OpenGL Performer Library for graphical
representation
Sound Rendering I
• Calculates velocity vector at the position of the virtual
microphone depending on interaction sphere radius
(using Schaeffer’s interpolation scheme):
• Small : from vertices of the grid cell
• Large: from all the vertices inside the influence sphere
• Velocity value & angle at that position
sForallNode
nm
sForallNode
nmn
m
rr
rrt
t2
2
/1
/)(
)(
p
p
Sound Rendering II
• Two output values for both angle and
velocity:
• Output = value / max value
• Output = (value / max value) 5/3
• Relationship between loudness level and
intensity: S ~ a3/5 [B.Gold]
• Thus, a function between values and
amplitude should be:
a = const * data value5/3
to imply: S ~ data value
)(tv
Virtual
Microphone
Direction
11
• Amplitude ~ * 5/3
• were v5/3-> [0,1]
• and a5/3-> [0.5, 1]
Sound Rendering III
Frequency ~
were v -> [0,1] -> [500, 1500]
3/5v
)(tv
)(tv
White band noise is modified in amplitude
and frequency to simulate a wind effect
Sonification Types
Positive vs. Negative Amplitude Modulation• velocity value is mapped to either increase or
decrease in amplitude of the sound
Amplitude vs. Frequency Modulation• highest velocity value is mapped to either
loudest noise or highest pitch noise
Before vs. after interpolation• many separate sounds for each vertex in the
local area vs. one sound of the interpolated value at the position of virtual pointer
Hypothesis
• According to multimodal theory adding sound rendering to visual rendering should improve exploration of CFD flow fields
• Testability: In our experiments, we will determine if our hypothesis is correct for an eddy localization task
• Simplicity: This hypothesis is simple and can easily be tested
• Null hypothesis: The combination of visual and auditory do not improve eddy localization efficiency.
Variables
Independent Variables
• Eddy localization in the flow field
• Starting point in the volume
• Sound mapping: frequency or amplitude modulation
• Sound rendering: positive or negative amplitude
modulation
• Interpolation type: Schaeffer’s or multi-sound
sources
• Virtual microphone radius R
Dependent Variables
• Eddy localization error
• Time and length of trajectory to get to the eddy
Interface Evaluation Procedure
1. Design the experiment.
2. Conduct the experiment.
3. Collect the data.
4. Analyze the data.
5. Draw your conclusions & establish
hypotheses
6. Redesign and do it again.
Usability Experimental Setup
Visual and/or Audio cues, haptic - navigation
12
Usability Study Experimental
Setup
Participants asked to locate vortex centers
40 fields: 25x25x25
• Random vortex locations
• Red arrow / specific sound
• 15 warm-up trials
• 36 experimental trials
• Random start point
• Random setup
Usability Study Results
Worst results for the audio-alone system:• participants are slower in locating the goal
• participants are less efficient in exploring the
volume
• Participants are less precise in locating the goal
Multimodal vs. Visual Only
Interface
Equal or better results for the multi-modal system
• Participants explore less space
• Participants are much faster in locating the goal position
Importance of Sound Rendering
Parameters
Specific system setup helps to improve performance
The Best Configuration:
Positive amplitude with
large radius and before
interpolation
Usability Study Conclusion
• The multimodal system is more
efficient to localize eddies than either
pure visual or pure audio systems
• Specific mapping parameters influence
system performance
• Different audio parameters are better
for audio-only than for multi-modal
interface
• Different audio parameters might be
better for different conditions
Multi-modal Exploration of Electric
Filds
Melenas 2010
13
Molecule Docking
Melenas 2010
Solar TErrestrial RElations
Observatory (STEREO)
NASA Picture
May, 2006
Pictures by Johns-Hopkins Applied Physics Laboratory
STEREO Data
Usable Data from STEREO
• Stereo Pairs of the sun in visible and ultra-violet bands
• 3-D Reconstructions of the Corona and the sun main body
• 3-D distribution of plasma characteristics of solar energetic particles
• Local vector magnetic field
• Plasma characteristics of protons, alpha particles and heavy ions
• Trace the generation and evolution of traveling radio disturbances
STEREO Multi-Modal Interface
System for Data Driven Simulations
NASA
STEREODatabase
Sun Corona
Simulation
Sun Particle
Emission
Simulation
Sun
Electro-Magnetic
Emission
Simulation
Multi-Modal Data
Exploration
Interface
Earth
Electro-Magnetic
Disturbances
Simulation