delivering immersive experiences - gtc on...

1www.esi-group.com

Copyright © ESI Group, 2017. All rights reserved.Copyright © ESI Group, 2017. All rights reserved.

www.esi-group.com

SAN JOSÉ | 08 – 11 MAY 2017

DELIVERING IMMERSIVE EXPERIENCESTHROUGH GPU VIRTUALIZATION AND STREAMING

Jan Wurster08 MAY 2017 | S7203 5:00pm to 5:50pm

2www.esi-group.com

Copyright © ESI Group, 2017. All rights reserved.

“A pioneer and world-leading provider in Virtual Prototyping.”

— www.esi-group.com

3www.esi-group.com


ENGINEERING WORKSPACEScourtesy of Getty Images

4www.esi-group.com


ENGINEERING WORKSPACEScourtesy of BMW Group

5www.esi-group.com


ENGINEERING WORKSPACES

6www.esi-group.com


ENGINEERING WORKSPACEScourtesy of Valmet Automotive

7www.esi-group.com

Copyright © ESI Group, 2017. All rights reserved.Copyright © ESI Group, 2017. All rights reserved.

“Augmented And Virtual Reality Fuel The Future Workplace“

Forbes Mag, Nov 2016

8www.esi-group.com


Evolution of Engineering Workspaces

Once upon a time: Physical Prototypes as part ofthe workspace

Interacting with peers

Continuous validation andpractical engineering

Established VR: Virtual Prototypes in Virtual Environments

All but removes the need for physical prototypes, saving time, effort and resources

Great for Collaboration!

Cost, complexity, maintenance and availability in digital prototypes sometimes lead to less validation andcollaboration.

9www.esi-group.com



Workplace VR! Finally: Affordable immersive hardware!

However .. Collaboration?

Efficient use of expensive (and noisy) resources?

Complexity of setup and maintenance?

Software affordabiblity?

10www.esi-group.com



11www.esi-group.com



12www.esi-group.com



• Offer the benefits of VR in engineering to non-experts

• Adopt to and support changes in infrastructure

• Integrate and support processes in the 4.0 enterprise

• Lower the technical obstacleswithout compromising quality ofresults

• Use the opportunity of mobile VR ubiquity - match today‘s flexible workstyles

SOFTWARE AFFORDABILITY

HARDWARE ACCESSIBILITY

INTUITIVE USABILITY

Virtualization& Cloud

Technology

Immersive

Virtual Reality Experience

13www.esi-group.com


Working Environments in Engineering

High Quality Immersive Virtual Reality to the desk

VR IN THE DATACENTER

Untether Virtual Reality experiences from the workspace

IMMERSIVE CLOUD EXPERIENCES AND MOBILE VR

Collaboratively Investigate, Evaluate and Decide

COLLABORATIVE IMMERSIVE SPACES

14www.esi-group.com


The Engineering Workspace

OBJECTIVES & REQUIREMENTS INFRASTRUCTURE

• Accessibility & Democratization• Virtual Reality at the Workplace –

without the technical complexity ofworkstation-based VR

• High Quality ImmersiveEnvironments (CAVE, Powerwall)

• Share Resources• Exploit available computing

resources more efficiently throughsharing

• Enterprise Visualization Datacenter• GPU virtualization to support thin /

mobile clients

• High quality streaming via High Bandwidth / low latency enterprisenetworks

• Building a GPU infrastructure in Datacenters based on NVIDIA GRID / Tesla infrastructure to support bothtraditional engineering and immersivevirtual reality applications

15www.esi-group.com


Thin Client Prototyping Challenges

HMD prototyping Scenario requires the lowest possible latency for full end to end roundtrip

at the highest feasible throughput

with the most challenging aspects of user comfort

Setup Vive - capable Workstation as a simplified backend

Dual M6000 NVIDIA GPUs

IC.IDO with Helios Renderer based on NVIDIA NV-Pro pipeline Talk: S7184: ENGINEERING DATASETS ON HMDS

Standard 1GBit Ethernet network

Standard-Issue low power Notebook as a thin client

Visit us atBooth 708

-Try the demo!

May 8-11, 2017 | Silicon Valley

16www.esi-group.com


Thin Client Prototyping Challenges

BANDWIDTH

LATENCY

INFRASTRUCTURE INTEGRATION

USER COMFORT

17www.esi-group.com


Headroomup to 40ms

Proc & Display 10 ms

Theory: Streaming Virtual Reality Content

HMD Scenario

HMD Scenario Throughput Target Rate 90 fps

45 fps (with device processing: warping/reprojection)

Motion to Photon Latency Imperceptable: <= 20ms

Acceptable: < 80ms

NOTE: Non-Gaming Scenario!

BANDWIDTHLATENCY

ApplicationRendering10-20 ms

Tracking5 ms

BackendRx, App,Visualize, Encode, Tx25ms

Thin ClientRx, Decode, Display20ms

Thin ClientInput & Tx8ms

Network2 ms

Network4 ms


Decode, Render5 ms

Encode3 ms

ApplicationRendering10-20 ms

Tracking5 ms

Streaming VR at a total latency below 80ms seems feasible.

18www.esi-group.com


Thin Client – Solution Architecture

Thin

Clie

nt

HELIOS

IC.I

DO

INST

AN

CE NVIDIA CAPTURE

SDKSTREAMING

NVENC NVIFR

STREAMING SERVER

NVIDIA GRIDvGPU

VCA

TRACKING

NVIDIA Codec

SDK

OpenVR

FRA

ME

REC

EIV

ER

GPU

INFRASTRUCTUREBANDWIDTH

19www.esi-group.com


Measurements: Streaming Virtual Reality Content

Backend: Dell T5810 Workstation 1x Quadro M6000

Thin Client: Dell Notebook NVIDIA Quadro

(Measurements averaged over several minutes of interactions)

BANDWIDTHLATENCY


NVDEC, Render8 ms

NVENC3 ms

App, Render8 ms

Tracking5 ms

Local 1GBase-T Network

Rx,Application,Rendering,

Encoding<12 ms

Tx, Rx (Frame) 10ms

Motion toPhoton Latency

~ 45ms

Latency Overhead forstreaming: <20ms

(tcp (websockets), NVENC, NVDEC)

20www.esi-group.com


Measurements: Streaming Virtual Reality Content

Results Throughput tested up to 250 fps

(1GBase-T local network)

Trading off throughput and latency toimage quality required!

90 fps are achievable

45 fps when combined with device-side processing can appear superiordue to higher quality encoding

BANDWIDTHLATENCY

0,00

10,00

20,00

30,00

40,00

50,00

60,00

70,00

80,00

90,00

100,00

90 fps (1GBase-T) 180 fps (1GBase-T) 80 fps (802.11ac)

Latency Measurementsmin max avg

21www.esi-group.com


Streaming Virtual Reality Content

Improving Headroom Trying to parallelize sending tracking data /

receiving frames / decoding & displaying frameson the client did not yield significantimprovements

Optimization of system latency should focus theapplication and visualization subsystems.

Avoiding and improving Network-side latency forframe data provides the biggest remainingpotential

Introduce Redundancy for stream rate stability in clogged / non-dedicated network protocols (oreven wireless)

USER COMFORT


NVDEC, Render8 ms

NVENC3 ms

App, Render8 ms

Tracking5 ms

Local 1GBase-T Network

App and Backend Image Generation

Network Protocol

Client Decoding andRendering

22www.esi-group.com


Further Work: Streaming Virtual Reality Content

Increasing User Comfort Composite Image Generation

Stream high complexity data

Render low complexity data locally

User Representation Introduce avatar representation (*) as low-latency elements rendered locally

Optimize Streaming Protocol Parallelized UDP streams, reintegrated based on first received packet

Stream Video Quality, Resolution Optimize streaming resolution to reflect properties of HMD optics

Optimize Codec and parameter choice

USER COMFORT

(*) Augmenting immersive cinematic experience/scene with user body visualizationChen, Joshua Yang, 2016, http://hdl.handle.net/10092/12144Nasum Virtualis: A Simple Technique for Reducing Simulator SicknessDavid Matthew Whittinghill, Bradley Ziegler, James Moore, and Tristan Case

23www.esi-group.com


On Premise / Enterprise Cloud Stack

NVIDIA GRID | HypervisorBare Metal

Conceptualizing an Immersive Datacenter Infrastructure

vGPU vGPUvGPU

GPUEnterprise Storage

Ceph, Swift, ..

DesktopApp

IC.IDOInstance(Master)

Thin ClientImmersiveSetups

Application & Visualization

VirtualizationDatacenter

INFRASTRUCTURE

vGPU

GPU

Enterprise Network


IDO.Cooperate Cluster

24www.esi-group.com





vGPU vGPUvGPUvGPU

GPUGPUEnterprise Storage

Ceph, Swift, ..


IC.IDOInstance(Slave)


Visualization Cluster

DesktopApp


Enterprise Network




INFRASTRUCTURE

Immersive Projection Setups

25www.esi-group.com





vGPU vGPUvGPUvGPU

GPUGPUEnterprise Storage

Ceph, Swift, ..




Visualization Cluster

vrifyCloudInstance


Enterprise Network




INFRASTRUCTURE

Immersive Projection Setups

IDO.Cooperate Cluster

26www.esi-group.com


Immersive Cloud Experiences & mobile VR

IndustrialDesigners

Design Engineers

Manufacturing& Integration

Engineers

Maintenance & repair

operators

Makers & Enthusiasts

Engineering Schools

Educators

27www.esi-group.com


Personal Immersive VR HMDs

Virtual Reality Technology Evolution

Projection-based VRCAVE, Powerwall

Co

llab

ora

tive

VR

Imm

ers

ive

VR

Mobile VRCardboard, GearVR, Daydream

mo

bile

VR

Come visitbooth 704 at

GTC 2017!

Enterprises & Institutes

Engineering & Research

Enterprises & Institutes

Recreation & Experiences

Gamers & Enthusiasts

Engineering & Research

ContentContent

Audience

AudienceEnterprises & Institutes

Gamers & Enthusiasts

„Everybody Else“

Audience

360° Videos, Games, ?Content

Established Virtual Reality SolutionsImmersive Virtual Engineering

28www.esi-group.com


Designing for mobile Virtual Reality

SCENARIO REQUIREMENT PLATFORM CHARACTERISTICS

• VR Rendering Paradigms• Low latency

• 60++ FPS

• Highly dynamic and interactive

• Engineering Scenario Characteristics• 20++ Million Triangles

• Physics Simulation

• Advanced Appearances

• Global Illumination

• Limited Device Resources

• Stereoscopic Viewpoint and Tracking • No positional head tracking

• Screen and Lens Quality• Low resolution and optical quality

29www.esi-group.com


Overcoming Limitations

COMPUTING POWER

GPU RESOURCES

INTERACTION CAPABILITIES

3DOF TRACKING

30www.esi-group.com


Overcoming Limitations

Decoupled Remote Visualization

Cloud Computing Infrastructure

Hybrid User Experience

31www.esi-group.com


Hybrid User ExperienceINTERACTION3DOF TRACKING

WEB

CLI

ENT

WEB

CLI

ENT

VR

WEB

CLI

ENT

Webbased client for Mobile VR Immersive Virtual Reality as a ‚second screen‘

Client-local User Interface - low latency and direct interaction

HTML5 - platform-agnostic , responsive UI

Flexible input and usability

WEB

SER

VIC

E

VIS

UA

LIZA

TIO

NSE

RV

ICE

Physics

Visualization

Application

AP

PLI

CA

TIO

N

SER

VIC

E

Auth

AP

PLI

CA

TIO

N

SER

VIC

E Auth

AP

PLI

CA

TIO

NSE

RV

ICE

Storage

Auth

Scaling

PUBLIC / PRIVATE CLOUD INFRASTRUCTURE

32www.esi-group.com


Hybrid User ExperienceINTERACTION3DOF TRACKING

WEB

CLI

ENT

WEB

CLI

ENT

VR

WEB

CLI

ENT

Engineering use cases require scaling in multiple dimensions Multi-User and Multi-Tenant

Compute capacity for realtime physics

Collaboration and sharing of results

WEB

SER

VIC

E

VIS

UA

LIZA

TIO

NSE

RV

ICE

Physics

Visualization

Application

AP

PLI

CA

TIO

N

SER

VIC

E

Auth

AP

PLI

CA

TIO

N

SER

VIC

E Auth

AP

PLI

CA

TIO

NSE

RV

ICE

Storage

Auth

Scaling

PUBLIC / PRIVATE CLOUD INFRASTRUCTURE

33www.esi-group.com


VIS

UA

LIZA

TIO

N S

ERV

ICE

VIS

UA

LIZA

TIO

N S

ERV

ICE

VR

WEB

CLI

ENT

Decoupled Remote Visualization PipelineGPU RESOURCES 3DOF TRACKING

OPTIX

OPENGL

VIS

UA

LIZA

TIO

N S

ERV

ICE

NVIDIA CAPTURESDK

STREAMING NVENC NVIFR

STREAMING SERVER

NVIDIA GRIDvGPU

VCA

SCALABLE NVIDIA GRID vGPU INFRASTRUCTURE

34www.esi-group.com


Rendering Stereoscopic Panoramics


LEFT

RIGHT

Top / Bottom Format at >= 2048x2048 resolution

Omnidirectional Stereo Low visualizuation latency

Feasible bandwidth for streaming

Interactive / dynamic content

Straightforward through Raytracing

Can be approximated throughRasterization

Plenoptic Rendering / Lightfields Some location independence

Static content

Synthesis Non-Realtime

35www.esi-group.com


ODS Image Synthesis

Rendering Stereoscopic Panoramics

STEREO CAMERA SETUP ODS APPROXIMATED ODS APPROXIMATED

ReferencesP.D. Bourke. “Synthetic Stereoscopic Panoramic Images”Google Developers – „Rendering Omni‐directional Stereo Content“

36www.esi-group.com


Displaying Stereoscopic Panoramics


Tracking & Projection

Warping & Lens Correction

Decoding

Immersive UI Overlay

WEB

CLI

ENT

STR

EAMWebVR Client for mobile devices

Device APIs for efficient GPU-enabledDecoding

Equirectangular imagestexturemapped to left / right eyespheres

Client-side User Interface Overlay

Device APIs for Warping and Lens Correction

Compatible to GearVR, Cardboard, Rift, Vive, Daydream

37www.esi-group.com


“Rendering Omni‐directional Stereo Content”— Source: Google Developers,https://developers.google.com/vr/jump/rendering-ods-content.pdf

“Synthetic Stereoscopic Panoramic Images”

— Source: P.D. Bourke. Lecture Notes in Computer Science (LNCS), Springer,

ISBN 978-3-540-46304-7, Volume 4270, 2006, pp 147-155

“Nasum Virtualis: A Simple Technique for Reducing Simulator Sickness”

— Source: David Matthew Whittinghill, Bradley Ziegler, James Moore, and Tristan Case,

http://www.purdue.edu/newsroom/releases/2015/Q1/virtual-nose-may-reduce-simulator-sickness-in-video-games.html

“Augmenting immersive cinematic experience/scene with user body visualization”

— Source: Chen, Joshua Yang, 2016, http://hdl.handle.net/10092/12144

38www.esi-group.com


Acknowledgements and Team

‣ NVIDIA DevTech Team

‣ Don Breda• NVIDIA – ESI Alliance Manager

‣ NVIDIA Capture• https://developer.nvidia.com/capture-sdk

‣ Andreas Kleber• HTC Vive Device Support• Oculus Rift Device Support

‣ Thomas Kipshagen• Streaming Client and Protocol• Optimizations

‣ Andreas Mank• HMD integration• Raytracing Omidirectional Stereo• Thin Client Threading

‣ Michael Hofmann• Integration with Helios and Porting

‣ Trevor Dodds• Streaming Server

39www.esi-group.com


THANK YOU

[email protected]

May 8-11, 2017 | Silicon Valley

http://icido.com http://vrify.esi-group.com

mailto:[email protected]

delivering immersive experiences - gtc on...

Documents