nvidia cloudxr - on-demand.gputechconf.comdecode hevc frame sensor & input data rgb frame audio...
TRANSCRIPT
![Page 1: NVIDIA CLOUDXR - on-demand.gputechconf.comDecode HEVC Frame Sensor & input data RGB Frame Audio er p Audio Capture / Encode k Frame Pacing Jitter FIFO CLIENT Display VR Runtime Controller](https://reader036.vdocument.in/reader036/viewer/2022071508/61299b055257731b08691489/html5/thumbnails/1.jpg)
David Weinstein, 19 December 2019
NVIDIA CLOUDXR™
![Page 2: NVIDIA CLOUDXR - on-demand.gputechconf.comDecode HEVC Frame Sensor & input data RGB Frame Audio er p Audio Capture / Encode k Frame Pacing Jitter FIFO CLIENT Display VR Runtime Controller](https://reader036.vdocument.in/reader036/viewer/2022071508/61299b055257731b08691489/html5/thumbnails/2.jpg)
2
Gaming
Movies
Sports
Concerts
Travel Retail
PERSONAL ENTERTAINMENT
VR
![Page 3: NVIDIA CLOUDXR - on-demand.gputechconf.comDecode HEVC Frame Sensor & input data RGB Frame Audio er p Audio Capture / Encode k Frame Pacing Jitter FIFO CLIENT Display VR Runtime Controller](https://reader036.vdocument.in/reader036/viewer/2022071508/61299b055257731b08691489/html5/thumbnails/3.jpg)
3NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.
QUADRO PROFESSIONAL VR/AR
MEDICAL ARCHITECTURE
DESIGN MANUFACTURING COLLABORATION
DIGITAL CONTENT CREATION
![Page 4: NVIDIA CLOUDXR - on-demand.gputechconf.comDecode HEVC Frame Sensor & input data RGB Frame Audio er p Audio Capture / Encode k Frame Pacing Jitter FIFO CLIENT Display VR Runtime Controller](https://reader036.vdocument.in/reader036/viewer/2022071508/61299b055257731b08691489/html5/thumbnails/4.jpg)
4
![Page 5: NVIDIA CLOUDXR - on-demand.gputechconf.comDecode HEVC Frame Sensor & input data RGB Frame Audio er p Audio Capture / Encode k Frame Pacing Jitter FIFO CLIENT Display VR Runtime Controller](https://reader036.vdocument.in/reader036/viewer/2022071508/61299b055257731b08691489/html5/thumbnails/5.jpg)
5
![Page 6: NVIDIA CLOUDXR - on-demand.gputechconf.comDecode HEVC Frame Sensor & input data RGB Frame Audio er p Audio Capture / Encode k Frame Pacing Jitter FIFO CLIENT Display VR Runtime Controller](https://reader036.vdocument.in/reader036/viewer/2022071508/61299b055257731b08691489/html5/thumbnails/6.jpg)
6
![Page 7: NVIDIA CLOUDXR - on-demand.gputechconf.comDecode HEVC Frame Sensor & input data RGB Frame Audio er p Audio Capture / Encode k Frame Pacing Jitter FIFO CLIENT Display VR Runtime Controller](https://reader036.vdocument.in/reader036/viewer/2022071508/61299b055257731b08691489/html5/thumbnails/7.jpg)
7
FUNDAMENTAL VR GRAPHICS CHALLENGES
TRADITIONAL = 60 MP/S(1920 X 1080 @ 30 FPS)
1920
1080
VIRTUAL REALITY = 450 MP/S(3024 X 1680* @ 90 FPS)
1512
1680
1512 Motion to Photon: ≤ 20 ms
Very High Pixel Rendering Throughput Must Maintain Very Low Latency
Pixel Throughput and Perceived Latency
HMD updates pose
CPU triggers redraw
GPU computes new image
HMD flashes display
![Page 8: NVIDIA CLOUDXR - on-demand.gputechconf.comDecode HEVC Frame Sensor & input data RGB Frame Audio er p Audio Capture / Encode k Frame Pacing Jitter FIFO CLIENT Display VR Runtime Controller](https://reader036.vdocument.in/reader036/viewer/2022071508/61299b055257731b08691489/html5/thumbnails/8.jpg)
8
WANTED: THE BEST OF BOTH WORLDS
Highest Quality Graphics + Mobile Freedom
![Page 9: NVIDIA CLOUDXR - on-demand.gputechconf.comDecode HEVC Frame Sensor & input data RGB Frame Audio er p Audio Capture / Encode k Frame Pacing Jitter FIFO CLIENT Display VR Runtime Controller](https://reader036.vdocument.in/reader036/viewer/2022071508/61299b055257731b08691489/html5/thumbnails/9.jpg)
9
CLOUDXR VISION
RTX ServersContent Network
Clients
XR Streaming to Any Device
![Page 10: NVIDIA CLOUDXR - on-demand.gputechconf.comDecode HEVC Frame Sensor & input data RGB Frame Audio er p Audio Capture / Encode k Frame Pacing Jitter FIFO CLIENT Display VR Runtime Controller](https://reader036.vdocument.in/reader036/viewer/2022071508/61299b055257731b08691489/html5/thumbnails/10.jpg)
10
MOBILE WORLD CONGRESS (LA), OCT 2019
CloudXR Announcement
![Page 11: NVIDIA CLOUDXR - on-demand.gputechconf.comDecode HEVC Frame Sensor & input data RGB Frame Audio er p Audio Capture / Encode k Frame Pacing Jitter FIFO CLIENT Display VR Runtime Controller](https://reader036.vdocument.in/reader036/viewer/2022071508/61299b055257731b08691489/html5/thumbnails/11.jpg)
11
ARCHITECTURE OVERVIEWServer-Side Driver, Client-Side Application and SDK
OpenVR Runtime (Valve)
VR App
Virtual HMD Driver
Cloud
Late & Lens Warp
VR Runtime
Video Decoder
Cloud
HMD
Video Encoder
Virtual Audio Driver
Audio Encoder Audio Decoder
InternetSERVER driver (Nvidia) CLIENT app (Nvidia)
audio, video, haptics
pose, control inputs, video
![Page 12: NVIDIA CLOUDXR - on-demand.gputechconf.comDecode HEVC Frame Sensor & input data RGB Frame Audio er p Audio Capture / Encode k Frame Pacing Jitter FIFO CLIENT Display VR Runtime Controller](https://reader036.vdocument.in/reader036/viewer/2022071508/61299b055257731b08691489/html5/thumbnails/12.jpg)
12
ARCHITECTUREWIN SERVER CLIENT
INST
ALL
ER
Audio/Video
Data (sensors, inputs, QoS)
Encode
RGB Frame
Display
VR App
Decode
Vid
eo P
ara
ms
(e.g
. RO
I, o
pti
on
al)
HEVC Frame HEVC Frame
VR Runtime
Controller & inputs
Sensor,Pose &Controller Data
RGB Frame
Sensor & input data
Sensor & input data
Audio
(e.g., OpenVR)
Ap
p
Clie
nt
app
Serv
er d
rive
r
Time Warp
Lens Warp
RGB Frame
Driver API
VR Runtime
Runtime API
OpenVR Interfaces
HMD
Au
dio
Pla
ybac
k
Audio Capture / Encode
![Page 13: NVIDIA CLOUDXR - on-demand.gputechconf.comDecode HEVC Frame Sensor & input data RGB Frame Audio er p Audio Capture / Encode k Frame Pacing Jitter FIFO CLIENT Display VR Runtime Controller](https://reader036.vdocument.in/reader036/viewer/2022071508/61299b055257731b08691489/html5/thumbnails/13.jpg)
13
QOS
Relevant Internet problems:
• Jitter
• Missed packets
• Bandwidth variation
Reliability & Resilience
![Page 14: NVIDIA CLOUDXR - on-demand.gputechconf.comDecode HEVC Frame Sensor & input data RGB Frame Audio er p Audio Capture / Encode k Frame Pacing Jitter FIFO CLIENT Display VR Runtime Controller](https://reader036.vdocument.in/reader036/viewer/2022071508/61299b055257731b08691489/html5/thumbnails/14.jpg)
14
QOS
Reliability & Resilience
Challenges Solutions Control parameters
Latency Custom HEVC profile NVENC knobs
Jitter De-jitter buffer Buffer size
Missed packets Forward error correction Correction strength
Bandwidth variation Video compression Compression rate
Server has no display Frame pacing to match
client display rate
Server render rate
![Page 15: NVIDIA CLOUDXR - on-demand.gputechconf.comDecode HEVC Frame Sensor & input data RGB Frame Audio er p Audio Capture / Encode k Frame Pacing Jitter FIFO CLIENT Display VR Runtime Controller](https://reader036.vdocument.in/reader036/viewer/2022071508/61299b055257731b08691489/html5/thumbnails/15.jpg)
15
WIN SERVER
VR App
Vid
eo P
ara
ms
(e.g
. RO
I, o
pti
on
al)
Ap
p
Driver API
VR Runtime
Runtime API
ARCHITECTURE
INST
ALL
ER
Audio / Video / Data (sensors, inputs, QoS)
Encode
RGB Frame
Decode
HEVC Frame
Sensor,Pose &Controller data
Sensor & input data
Audio
Clie
nt
app
Serv
er d
rive
r
Audio Capture / Encode
Au
dio
Pla
ybac
k
Frame Pacing
Jitter FIFO
CLIENT
Display
VR Runtime
Controller & inputs
(e.g., OpenVR)Time Warp
Lens Warp
HMD
OpenVR Interfaces
![Page 16: NVIDIA CLOUDXR - on-demand.gputechconf.comDecode HEVC Frame Sensor & input data RGB Frame Audio er p Audio Capture / Encode k Frame Pacing Jitter FIFO CLIENT Display VR Runtime Controller](https://reader036.vdocument.in/reader036/viewer/2022071508/61299b055257731b08691489/html5/thumbnails/16.jpg)
16
LATENCY
Using:
• HelloVR sample modified to change color on button press
• Latency meter (instrumented switch + photodiode)
• Local Cloud with Client on WiFi (average ping time 4 ms)
Button to Photons
![Page 17: NVIDIA CLOUDXR - on-demand.gputechconf.comDecode HEVC Frame Sensor & input data RGB Frame Audio er p Audio Capture / Encode k Frame Pacing Jitter FIFO CLIENT Display VR Runtime Controller](https://reader036.vdocument.in/reader036/viewer/2022071508/61299b055257731b08691489/html5/thumbnails/17.jpg)
17
LATENCY
Results:
Server Driver + Client App latency = 12.1 ms.
The Jitter FIFO can be downsized at the cost of occasional missed frames.
Button to Photons
Latency [ms] Stdev
Local VR 38.3 3.4
CloudXR 81.6 3.3
Diff 43.3 -0.1
Jitter FIFO + Ping 31.2
Diff – Above 12.1
*Average of 40 clicks
![Page 18: NVIDIA CLOUDXR - on-demand.gputechconf.comDecode HEVC Frame Sensor & input data RGB Frame Audio er p Audio Capture / Encode k Frame Pacing Jitter FIFO CLIENT Display VR Runtime Controller](https://reader036.vdocument.in/reader036/viewer/2022071508/61299b055257731b08691489/html5/thumbnails/18.jpg)
18
WIN SERVER
VR App
Vid
eo P
ara
ms
(e.g
. RO
I, o
pti
on
al)
Ap
p
Driver API
VR Runtime
Runtime API
LATENCY
INST
ALL
ER
Audio / Video / Data (sensors, inputs, QoS)
Encode
RGB Frame
Decode
HEVC Frame
RGB FrameSensor & input data
Audio
Clie
nt
app
Serv
er d
rive
r
Audio Capture / Encode
Au
dio
Pla
ybac
k
Frame Pacing
Jitter FIFO
CLIENT
Display
VR Runtime
Controller & inputs
(e.g., OpenVR)Time Warp
Lens Warp
HMD
3-10 ms
10-50 ms(depending on network)
1-3 ms
3-10 ms
OpenVR Interfaces < 0.5 ms
![Page 19: NVIDIA CLOUDXR - on-demand.gputechconf.comDecode HEVC Frame Sensor & input data RGB Frame Audio er p Audio Capture / Encode k Frame Pacing Jitter FIFO CLIENT Display VR Runtime Controller](https://reader036.vdocument.in/reader036/viewer/2022071508/61299b055257731b08691489/html5/thumbnails/19.jpg)
19
![Page 20: NVIDIA CLOUDXR - on-demand.gputechconf.comDecode HEVC Frame Sensor & input data RGB Frame Audio er p Audio Capture / Encode k Frame Pacing Jitter FIFO CLIENT Display VR Runtime Controller](https://reader036.vdocument.in/reader036/viewer/2022071508/61299b055257731b08691489/html5/thumbnails/20.jpg)
20
Total Flexibility
CLOUDXR CONFIGURATIONS
5G
Ethernet
WiFi
![Page 21: NVIDIA CLOUDXR - on-demand.gputechconf.comDecode HEVC Frame Sensor & input data RGB Frame Audio er p Audio Capture / Encode k Frame Pacing Jitter FIFO CLIENT Display VR Runtime Controller](https://reader036.vdocument.in/reader036/viewer/2022071508/61299b055257731b08691489/html5/thumbnails/21.jpg)
21
EARLY ACCESS PROGRAMTest CLOUDXR for VR
@ https://developer.nvidia.com/nvidia-cloudxr-sdk-early-access-program
![Page 22: NVIDIA CLOUDXR - on-demand.gputechconf.comDecode HEVC Frame Sensor & input data RGB Frame Audio er p Audio Capture / Encode k Frame Pacing Jitter FIFO CLIENT Display VR Runtime Controller](https://reader036.vdocument.in/reader036/viewer/2022071508/61299b055257731b08691489/html5/thumbnails/22.jpg)
22
NVIDIA 5G MEC XR DEVKIT
![Page 23: NVIDIA CLOUDXR - on-demand.gputechconf.comDecode HEVC Frame Sensor & input data RGB Frame Audio er p Audio Capture / Encode k Frame Pacing Jitter FIFO CLIENT Display VR Runtime Controller](https://reader036.vdocument.in/reader036/viewer/2022071508/61299b055257731b08691489/html5/thumbnails/23.jpg)
23
THE OPPORTUNITY: VR/AR/VGPU OVER 5GA Convergence of Visual Compute + 5G and a New Edge Topology
Powerful GPUsIncredible compute capabilities redefining
graphics
Bandwidth and Low LatencyBoth are required for VR/AR
Compute is at the EdgeVery few hops – Low Latency
![Page 24: NVIDIA CLOUDXR - on-demand.gputechconf.comDecode HEVC Frame Sensor & input data RGB Frame Audio er p Audio Capture / Encode k Frame Pacing Jitter FIFO CLIENT Display VR Runtime Controller](https://reader036.vdocument.in/reader036/viewer/2022071508/61299b055257731b08691489/html5/thumbnails/24.jpg)
24
ENTERPRISE MARKETSSuccess Across Verticals
ManufacturingEngineering, Design, Marketing/Sales
AECEntire Design Process,
Sales, Moving to Construction
M&EVirtual Production
TrainingPossibly the Largest
Market
Enscape
Nurulize IC.IDO
Autodesk
Location-Based EntertainmentEnterprise/Consumer
ESI
![Page 25: NVIDIA CLOUDXR - on-demand.gputechconf.comDecode HEVC Frame Sensor & input data RGB Frame Audio er p Audio Capture / Encode k Frame Pacing Jitter FIFO CLIENT Display VR Runtime Controller](https://reader036.vdocument.in/reader036/viewer/2022071508/61299b055257731b08691489/html5/thumbnails/25.jpg)
25
ENTERPRISE AR
DIGITAL MODELS ANNOTATEDINSTRUCTION/COLLABORATION
Driving Value Propositions
![Page 26: NVIDIA CLOUDXR - on-demand.gputechconf.comDecode HEVC Frame Sensor & input data RGB Frame Audio er p Audio Capture / Encode k Frame Pacing Jitter FIFO CLIENT Display VR Runtime Controller](https://reader036.vdocument.in/reader036/viewer/2022071508/61299b055257731b08691489/html5/thumbnails/26.jpg)
26
THE FUTURE OF MOBILE XR
• Ultra Low Latency
• High Bandwidth
• Cutting-edge Compute
• Cost Effective Delivery
• Valuable Use Cases
Requirements:
5G + VR/AR + vGPU at the Edge
![Page 27: NVIDIA CLOUDXR - on-demand.gputechconf.comDecode HEVC Frame Sensor & input data RGB Frame Audio er p Audio Capture / Encode k Frame Pacing Jitter FIFO CLIENT Display VR Runtime Controller](https://reader036.vdocument.in/reader036/viewer/2022071508/61299b055257731b08691489/html5/thumbnails/27.jpg)
27
RTX Performance in a Server
• Powered by Quadro RTX 8000/6000
• Ray Traced Global Illumination up to 96 GB Scenes
• Remoting and Multi-GPU Virtualization with Quadro virtual Datacenter Work Station (vDWS)
• All the technical advantages of Turing
NVIDIA RTX SERVER
![Page 28: NVIDIA CLOUDXR - on-demand.gputechconf.comDecode HEVC Frame Sensor & input data RGB Frame Audio er p Audio Capture / Encode k Frame Pacing Jitter FIFO CLIENT Display VR Runtime Controller](https://reader036.vdocument.in/reader036/viewer/2022071508/61299b055257731b08691489/html5/thumbnails/28.jpg)
28
XR/VGPU 5G EDGE STACKSoftware/Hardware Stack Demands a Diverse Effort
Cloud Orchestration
Containerized Edge Applications
Low Latency Streaming
Visual Computing and AI
5G transmission
5G Network
Virtualization
Enabling SDKs
Fundamental GPUDrivers
![Page 29: NVIDIA CLOUDXR - on-demand.gputechconf.comDecode HEVC Frame Sensor & input data RGB Frame Audio er p Audio Capture / Encode k Frame Pacing Jitter FIFO CLIENT Display VR Runtime Controller](https://reader036.vdocument.in/reader036/viewer/2022071508/61299b055257731b08691489/html5/thumbnails/29.jpg)
29
COME TRY THE DEMO ®ISTER ONLINE FOR EARLY ACCESS
https://developer.nvidia.com/nvidia-cloudxr-sdk-early-access-program
Visit the VR Village on the Expo Floor
![Page 30: NVIDIA CLOUDXR - on-demand.gputechconf.comDecode HEVC Frame Sensor & input data RGB Frame Audio er p Audio Capture / Encode k Frame Pacing Jitter FIFO CLIENT Display VR Runtime Controller](https://reader036.vdocument.in/reader036/viewer/2022071508/61299b055257731b08691489/html5/thumbnails/30.jpg)