developing a system for real-time numerical simulation ... › gtc › 2014 › ...simulation during...
TRANSCRIPT
-
ni.com
Developing a System for Real-Time Numerical Simulation during Physical Experiments in a
Wave Propagation LaboratoryMarch 27, 2014
Darren Schmidt (presenter)Lothar WenzelNI Scientific Research & Big Physics
Dr. Thomas BlumProf. Stewart GreenhalghProf. Johan RobertssonDr. Dirk van ManenMarlies VasmelETH-Zurich – D-EDRW – Geophysics
-
2ni.com
NI Team• Barry Hutt• Darren Schmidt• Balazs Toth• Lothar Wenzel
ETH Team• Christoph Baerlocher• Dr. Thomas Blum• Prof. Stewart Greenhalgh• Heinrich Horstmeyer• Prof. Johan Robertsson• Dr. Dirk van Manen• Marlies Vasmel• `
The International Team
www.ni.com www.geophysics.ethz.ch
http://www.ni.com/http://www.geophysics.ethz.ch/
-
3ni.com
Problem: Existing Laboratory Limitations
• Size of physical experiments limit the study of longer wavelengths or lower frequencies
Rock
reflection
• Boundaries of physical experiments introduce undesired reflections
-
4ni.com
Goals
• A new lab needs to:
• Address existing limitations
• Avoid introducing new barriers
• Allow for applications outside of geophysics
-
5ni.com
Solution – Virtualize the Physical Experiment
“Immerse the physical experiment in a live real-time 3D simulation much like a person wearing a virtual reality suit is plugged into a visual
simulation.”
Rock
Simulated Environment
-
6ni.com
Solution – Virtualize the Physical Experiment
“Immerse the physical experiment in a live real-time 3D simulation much like a person wearing a virtual reality suit is plugged into a visual
simulation.”
-
7ni.com
Solution – Virtualize the Physical Experiment
This new system allows
• Investigation of larger wavelengths
• Full control over wave propagation
Example: ‘Noise’ cancellation for undesired reflections.
-
8ni.com
Solution – Virtualize the Physical Experiment
This new system allows
• Investigation of larger wavelengths
• Full control over wave propagation
Example: ‘Noise’ cancellation for undesired reflections.
“hear”
-
9ni.com
Solution – Virtualize the Physical Experiment
This new system allows
• Investigation of larger wavelengths
• Full control over wave propagation
Example: ‘Noise’ cancellation for undesired reflections.
“cancel”
-
10ni.com
Solution – Virtualize the Physical Experiment
This new system allows
• Investigation of larger wavelengths
• Full control over wave propagation
Example: ‘Noise’ cancellation for undesired reflections.
“cancel”
-
11ni.com
How?
The physical components of the experiment are:• A 1.5m cube rock sample
1.5m
1.5m
1.5m
Rock
-
12ni.com
How?
The physical components of the experiment are:• A 1.5m cube rock sample
• A 2m cube of water containing the rock sample
2m
2m
2m
Water
-
13ni.com
How?
The physical components of the experiment are:• A 1.5m cube rock sample
• A 2m cube of water containing the rock sample
• 1000+ sensors on or close to rock surface
o Sensors are equally distributed across sixrecording surfaces
o Each sensor measures pressure
sensors
-
14ni.com
How?
The physical components of the experiment are:• A 1.5m cube rock sample
• A 2m cube of water containing the rock sample
• 1000+ sensors on or close to rock surface
o Sensors are equally distributed across sixrecording surfaces
o Each sensor measures pressure
• 1000+ actuators on or close to water surface
o Actuators are equally distributed across sixemitting surfaces
o Each actuator generates acoustic waves
actuators
-
15ni.com
How?
The physical components of the experiment are:• A 1.5m cube rock sample
• A 2m cube of water containing the rock sample
• 1000+ sensors on or close to rock surface
o Sensors are equally distributed across sixrecording surfaces
o Each sensor measures pressure
• 1000+ actuators on or close to water surface
o Actuators are equally distributed across sixemitting surfaces
o Each actuator generates acoustic waves
-
16ni.com
How?
The simulation part of the experiment involves:
• Acquisition: 1000+ channels• Data represents pressure
• 16-bit resolution
-
17ni.com
How?
The simulation part of the experiment involves:
• Acquisition: 1000+ channels
• Computation: Large PDE solver• Based on a pre-computed Green’s functions (two 1K x 1K x 250 ‘matrices’)
• Formulation requires pressure and velocity data
-
18ni.com
How?
The simulation part of the experiment involves:
• Acquisition: 1000+ channels
• Computation: Large PDE solver
• Control: 1000+ channels• Response represents ‘generator’ values
• 16-bit resolution
-
19ni.com
How?
The simulation part of the experiment involves:
• Acquisition: 1000+ channels
• Computation: Large PDE solver
• Control: 1000+ channels
• Real-time constraint: 50 ms cycle time• Dictated by the 20 kHz sampling rate
• Includes a complete acquirecomputecontrol cycle
-
20ni.com
Why …. Water?
So the system has the …
• Time to act
-
21ni.com
Why …. Water?
So the system has the …
• Time to act• Distance from recording to emitting surface = 25 cm
• Propagation time in water ~ 170 ms
Slo
we
r W
ave
Pro
pag
atio
n
-
22ni.com
Why …. Water?
So the system has the …
• Time to act
• Ability to react
Slo
we
r W
ave
Pro
pag
atio
n
-
23ni.com
Why …. Water?
So the system has the …
• Time to act
• Ability to react• Impedance differential impacts actuator performance
• Wave ‘power’ in water is ‘stronger’ than air
Slo
we
r W
ave
Pro
pag
atio
n
-
24ni.com
Why …. Water?
So the system has the …
• Time to act
• Ability to react• Impedance differential impacts actuator performance
• Wave ‘power’ in water is ‘stronger’ than air
Slo
we
r W
ave
Pro
pag
atio
n
-
25ni.com
Why …. Water?
So the system has the …
• Time to act
• Ability to react
• Knowledge to respond• Wave propagation in water is well-understood
• Defined by ‘computable’ Green’s functions
-
26ni.com
Why …. Water?
So the system has the …
• Time to act
• Ability to react
• Knowledge to respond• Wave propagation in water is well-understood
• Defined by ‘computable’ Green’s functions
-
27ni.com
Why … the Rock?
• Predicting wave propagation in water is ‘easy’• Behavior is known
• Models are based on linear operators
• Many computational optimizations are possible
-
28ni.com
Why … the Rock?
• Predicting wave propagation in water is ‘easy’
• Predicting wave propagation in rock is ‘hard’• Behavior is under exploration
• Models involve inherently non-linear functions
• Computational complexity is high
-
29ni.com
Why … the Rock?
• Predicting wave propagation in water is ‘easy’
• Predicting wave propagation in rock is ‘hard’
• As a result, this real-time system:• Leaves the nonlinear behavior to Mother Nature
• Captures the results at the recording surfaces
• Manipulates the physical experiment using known models
-
30ni.com
Why … Pressure & Velocity?
• Dictated by the physics (i.e. the Green’s functions)
-
31ni.com
Why … Pressure & Velocity?
• Dictated by the physics (i.e. the Green’s functions)
velocity pressure
-
32ni.com
Why … Pressure & Velocity?
• Dictated by the physics (i.e. the Green’s functions)
• How Do We Get Velocity?
pressurevelocity
-
33ni.com
Option 1: Duplicate Recording Surfaces• Benefit
o Compute velocity using six additional recording surfaces
o New sensors measure pressure along the surface normal
Challenge 1 – How Do We Get Velocity?
= sensor (for pressure)
= sensor (for velocity)
-
34ni.com
Option 1: Duplicate Recording Surfaces• Benefit
o Compute velocity using six additional recording surfaces
o New sensors measure pressure along the surface normal
• Work
o Compute the velocity using common difference method
Challenge 1 – How Do We Get Velocity?
= sensor (for pressure)
= sensor (for velocity)
-
35ni.com
Option 1: Duplicate Recording Surfaces• Benefit
o Compute velocity using six additional recording surfaces
o New sensors measure pressure along the surface normal
• Work
o Compute the velocity using common difference method
• Issue
o Sensor data is double and computation is increased.
Challenge 1 – How Do We Get Velocity?
= sensor (for pressure)
= sensor (for velocity)
-
36ni.com
Option 1: Duplicate Recording Surfaces
Option 2: Duplicate Surfaces Using Staggered Sensor Layout• Benefit
o Sensor data is not doubled
o Gaps in one recording surface are filled in the other
o Layout still leverages the surface normal
Challenge 1 – How Do We Get Velocity?
= sensor (for pressure)
= sensor (for velocity)
-
37ni.com
Option 1: Duplicate Recording Surfaces
Option 2: Duplicate Surfaces Using Staggered Sensor Layout• Benefit
o Sensor data is not doubled
o Gaps in one recording surface are filled in the other
o Layout still leverages the surface normal
• Work
o Reconstruct all missing pressure data using interpolation
Challenge 1 – How Do We Get Velocity?
= virtual sensor (for pressure)
= virtual sensor (for velocity)
= sensor (for pressure)
= sensor (for velocity)
-
38ni.com
Option 1: Duplicate Recording Surfaces
Option 2: Duplicate Surfaces Using Staggered Sensor Layout• Benefit
o Sensor data is not doubled
o Gaps in one recording surface are filled in the other
o Layout still leverages the surface normal
• Work
o Reconstruct all missing pressure data using interpolation
• Issue
o Computation increases.
Challenge 1 – How Do We Get Velocity?
= virtual sensor (for pressure)
= virtual sensor (for velocity)
= sensor (for pressure)
= sensor (for velocity)
-
39ni.com
Option 1: Duplicate Recording Surfaces
Option 2: Duplicate Surfaces Using Staggered Sensor Layout
Option 3: Optimize the Mathematical Model• Benefit
o Staggered layout is accounted for in the numerical simulation
Challenge 1 – How Do We Get Velocity?
= sensor (for pressure)
= sensor (for velocity)
-
40ni.com
Option 1: Duplicate Recording Surfaces
Option 2: Duplicate Surfaces Using Staggered Sensor Layout
Option 3: Optimize the Mathematical Model• Benefit
o Staggered layout is accounted for in the numerical simulation
• Work
o Velocity is implicitly computed via a new Green’s function
Challenge 1 – How Do We Get Velocity?
= sensor (for pressure)
= sensor (for velocity)
-
41ni.com
Option 1: Duplicate Recording Surfaces
Option 2: Duplicate Surfaces Using Staggered Sensor Layout
Option 3: Optimize the Mathematical Model• Benefit
o Staggered layout is accounted for in the numerical simulation
• Work
o Velocity is implicitly computed via a new Green’s function
• Issue
o No physical meaning to the new simulation model
Challenge 1 – How Do We Get Velocity?
= sensor (for pressure)
= sensor (for velocity)
-
42ni.com
Option 3 – Optimize the Mathematical Model
-
43ni.com
Option 3 – Optimize the Mathematical Model
pressurevelocity
-
44ni.com
Option 3 – Optimize the Mathematical Model
pressure
-
45ni.com
Option 3 – Optimize the Mathematical Model
pressure
computed in advance
-
46ni.com
Option 3 – Optimize the Mathematical Model
The new Green’s function:• Avoids increasing the sensor data• Reduces the computation by 50%
-
47ni.com
Challenge 2 – How to Simulate in Real-Time?
Given a cycle time of 50 ms
• Latency
• Bandwidth
-
48ni.com
Challenge 2 – How to Simulate in Real-Time?
Given a cycle time of 50 ms
• Latencyo Is an issue for data movement within the system
Ethernet USB PCI PCI Express
Bandwidth (MB/s) 125 (Gigabit) 600 (SuperSpeed) 132 4000 (x16)
Latency (ms) 1000 (Gigabit)
-
49ni.com
Challenge 2 – How to Simulate in Real-Time?
Given a cycle time of 50 ms
• Latencyo Is an issue for data movement within the system
o PCI Express bus offers high throughput and low latency
o Transfers are synchronized using FPGA technology
Ethernet USB PCI PCI Express
Bandwidth (MB/s) 125 (Gigabit) 600 (SuperSpeed) 132 4000 (x16)
Latency (ms) 1000 (Gigabit)
-
50ni.com
Challenge 2 – How to Simulate in Real-Time?
Given a cycle time of 50 ms
• Latency
• Bandwidth
o Is an issue for the Green’s functions computation
-
51ni.com
Challenge 2 – How to Simulate in Real-Time?
Given a cycle time of 50 ms
• Latency
• Bandwidth
o Is an issue for the Green’s functions computation
o Pre-computed Green’s functions data is large– 1000 sensors & 1000 actuators 1K x 1K
– Experiment runs for 250 time steps 250
-
52ni.com
Challenge 2 – How to Simulate in Real-Time?
Given a cycle time of 50 ms
• Latency
• Bandwidth
o Is an issue for the Green’s functions computation
o Pre-computed Green’s functions data is large– 1000 sensors & 1000 actuators 1K x 1K
– Experiment runs for 250 time steps 250
1000 {1K x (1K x 250)} sgemv ops
-
53ni.com
Challenge 2 – How to Simulate in Real-Time?
Given a cycle time of 50 ms
• Latency
• Bandwidth
o Is an issue for the Green’s functions computation
o Pre-computed Green’s functions data is large
o Consume 4GB of memory
Storage = 4GB
-
54ni.com
Challenge 2 – How to Simulate in Real-Time?
Given a cycle time of 50 ms
• Latency
• Bandwidth
o Is an issue for the Green’s functions computation
o Pre-computed Green’s functions data is large
o Consume 4GB of memory
o Computation requires 100TB/s memory bandwidth– Estimated with data movement overhead of 10 ms
System bandwidth = 100TB/s
-
55ni.com
Challenge 2 – How to Simulate in Real-Time?
Given a cycle time of 50 ms
• Latency
• Bandwidth
o Is an issue for the Green’s functions computation
o Pre-computed Green’s functions data is large
o Consume 4GB of memory
o Computation requires 100TB/s memory bandwidth
o To perform the computation in real-time
– Leverage multiple NVIDIA Tesla K40 GPUs
– Connected via PCI Express
– Optimize the Green’s functions implementation to reducethe number of GPUs required by an order of magnitude
Telsa K40: 288GB/s
-
56ni.com
GPUs
PCIe Switch
Recording surface
Emitting surface
PXImc
MXI Gen2
MXI Gen2
LabVIEW Timing/Synch
PXI FPGA
Network over PCIe GPU acceleration
Real-Time 3D Simulator Components
PCIe enclosures PCIe switches
-
57ni.com
Demo: System Simulation & Green’s Functions Computation
Simulation of the System
Sensor Data
Green’s Functions Computation
-
58ni.com
Special Thanks
NVIDIA
• Jerry Chen
• Cliff Woolley
• Duncan Poole
One Stop Systems
• Jaan Mannik
Dell
• Aron Bowan
-
59ni.com
References
• Vasmel et al (2013). “Immersive experimentation in a wave propagation laboratory,” J. Acoust. Soc. Am. 134, EL492-EL498.http://www.geos.ed.ac.uk/homes/acurtis/Vasmel_etal_JASA_2013.pdf
• Big Physics at National Instrument’s http://www.ni.com/physics/
http://www.geos.ed.ac.uk/homes/acurtis/Vasmel_etal_JASA_2013.pdfhttp://www.geos.ed.ac.uk/homes/acurtis/Vasmel_etal_JASA_2013.pdfhttp://www.ni.com/physics/