scalable remote rendering with augmented streaming

Scalable Remote Rendering with Augmented Streaming

Dawid Pająk, Robert Herzog, Elmar Eisemann, Karol Myszkowski, Hans-Peter Seidel

2

Motivation

Service oriented Hardware and location

independent computing Great for thin clients Code and confidential

data kept in a secure remote environment

Low-cost of maintenance Full-time access

3

Cloud Computing and Rendering

NVidia RealityServer & iRay

4

Cloud Computing and Rendering

5

MotivationServer

InternetInternet

Video Encoding

Client

Full-frameRendering

CG application

Video Decoding

Bandwidth:2-6Mbit per client

6

MotivationServer

InternetInternet

Video Decoding

Video Encoding

Clients

Full-frameRendering

CG applicationVideo

Decoding


7

MotivationServer

InternetInternet

Video Decoding

Video Encoding

Clients

Full-frameRendering

CG application

Video Decoding

Out of resources!


Design similar to current commercial solutions

Video Decoding

8

The Idea

Server

InternetInternet

Video Encoding

Clients

Low-resolutionFrame Rendering

CG application

Auxiliary StreamEncoding

Similar bandwidth:2-6Mbit per client

Video Decoding

Auxiliary Stream

Decoding

Upsampling

Video Decoding

Auxiliary Stream

Decoding

Upsampling

Video Decoding

Auxiliary Stream

Decoding

Upsampling

9

Motivation

Are thin clients really „thin”? 10 years ago – YES! Now – definitely NOT!

Year: 2000CPU: ARM7 33-75MhzNo FPUNo GPULow-res monochrome display

Year: 2006CPU: ARM11 333MhzFPU, SIMD capableGPU: OpenGL ES 1.1Display: 240x320 24-bit

Year: 2010CPU: Cortex A8 ~1GhzGPU: OpenGL ES 2.0Display: 640x960

Year: 2011 and upCPU: Dual/Quad CoreGPU: multi-core designHD-ready resolution display

10

Our solution

Server

InternetInternet

Video Encoding

Clients

Low-resolutionFrame Rendering

CG application

Auxiliary StreamEncoding

Similar bandwidth:2-6Mbit per client

Video Decoding

Auxiliary Stream

Decoding

Upsampling

Video Decoding

Auxiliary Stream

Decoding

Upsampling

Video Decoding

Auxiliary Stream

Decoding

Upsampling

11

Client

Client-side extra applicationsVideo

Decoding

Auxiliary Stream

Decoding

2x upsampled video usingdepth/motion encoded with H.264

Ours (same bandwidth)

Spatio-temporalUpsampling1

1[Herzog et al. 2010]

12

Our solution – server-side

RendererRenderer

High-res.attributebuffers


Low-res.jittered frame


H.264EncoderH.264

EncoderCameramatrix

Cameramatrix

Client side

13

Edgedetection

Edgedetection


RendererRenderer



H.264EncoderH.264

EncoderCameramatrix

Cameramatrix



Client side

14

Client side


RendererRenderer



H.264EncoderH.264

EncoderCameramatrix

Cameramatrix

Edgedetection

Edgedetection



J2K ContrastTransducer

J2K ContrastTransducer

Edge visual thresholding

15


RendererRenderer



H.264EncoderH.264

EncoderCameramatrix

Cameramatrix

Edgedetection

Edgedetection



QuantizationQuantization

• float→int conversion• non-linear mapping of values• depth – 10 bits• motion – 2x8 bits

• float→int conversion• non-linear mapping of values• depth – 10 bits• motion – 2x8 bits

Client side

16


RendererRenderer



H.264EncoderH.264

EncoderCameramatrix

Cameramatrix

Edgedetection

Edgedetection

QuantizationQuantization Edge imagediffusion

Edge imagediffusion



Client side

t-1

17

RendererRenderer






H.264EncoderH.264

EncoderCameramatrix

Cameramatrix

Edgedetection

Edgedetection


Edge imagediffusion

Client side

Push Pull

18

Client side


Edge imagediffusion

Edge imagediffusion

Push Pull

19


Edge imagediffusion

Edge imagediffusion

Client side

Push Pull

20


Edge imagediffusion

Edge imagediffusion

Client side

Push Pull

21


Edge imagediffusion

Edge imagediffusion

Client side

Push Pull

22


Edge imagediffusion

Edge imagediffusion

Client side

Push Pull

23


Edge imagediffusion

Edge imagediffusion

Client side

Push Pull

24


Edge imagediffusion

Edge imagediffusion

Client side

Push Pull

25


Edge imagediffusion

Edge imagediffusion

Client side

Push Pull

26

Client side


RendererRenderer



H.264EncoderH.264

EncoderCameramatrix

Cameramatrix

Edgedetection

Edgedetection


Edge imagediffusion



Diffusion error with respect to ground truth depth buffer

PSNR: 52.42dB PSNR: 48.47dB PSNR: 52.49dB

27

Client side


RendererRenderer



H.264EncoderH.264

EncoderCameramatrix

Cameramatrix

Edgedetection

Edgedetection


Edge imagediffusion

Current frameprediction




2[Didyk et al. 2010]

Grid-based warping2

28

Client side


RendererRenderer



H.264EncoderH.264

EncoderCameramatrix

Cameramatrix

Edgedetection

Edgedetection


Edge imagediffusion



Edge imageEncoder

Edge imageEncoder



Binary edge image Depth and motion residuals

Auxiliary data frame packet

Lossless AC Prediction + Quantization + AC

Edge samplessign

29

Binary edge image coding (edge topology)

Example:

Scanline traversal order

avg. bits per symbol

no edge (0), edge (1)symbol counters

compression ratio

30


x

x

xx

Samples unknown to the decoder!Samples unknown to the decoder!

• Probabilities for symbols depend not only on their frequencies, but also on the neighborhood (context)

Scanline traversal order

31

…

0 0 0 0

0 0 0 0

0 0 0 0

…


• Progressive encoding – pass 0/3

0 0 0

0 x

Samples taken from current frame predictionSamples taken from current frame prediction

32

0

1

0

1

0

1

0

1

0

1

0

1

0

1

0

1

0

1

0

1

0

1

0

1

……



33

……

0 2

1

0 2

1

0 2

1

0 2

1

0 2

1

0 2

1

0 2

1

0 2

1

0 2

1

0 2

1

0 2

1

0 2

1



34

……

0 2

3 1

0 2

3 1

0 2

3 1

0 2

3 1

0 2

3 1

0 2

3 1

0 2

3 1

0 2

3 1

0 2

3 1

0 2

3 1

0 2

3 1

0 2

3 1



35

Current frame prediction

Depth and motion value coding

Current frame

Predictor choice based on the minimum residual “history”Predictor choice based on the minimum residual “history”

?

36

RendererRenderer



H.264EncoderH.264

EncoderCameramatrix

Cameramatrix

Edgedetection

Edgedetection

QuantizationQuantization

Edge imageEncoder

Edge imageEncoder

Edge imagediffusion

Edge imagediffusion





36

Client side


3737

Client side

Our solution – client-side

Upsampling+

Other applications

Upsampling+

Other applications



H.264DecoderH.264

DecoderCameramatrix

Cameramatrix

Edge imageDecoder

Edge imageDecoder

Edge imagediffusion

Edge imagediffusion

Future frameprediction

Future frameprediction



38

Results – QualityPS

NR

[dB]

Slightly lower depth qualitySlightly lower depth quality

Still better reconstruction!Still better reconstruction!

• x264 software used for H.264 low-res. stream encoding (@2MBit)

• 2x2 upsampling ratio

39

Results – speed / scalabilityFramerate [FPS]

• x264 software used for H.264 low-res. stream encoding (@2MBit)

• 4x4 upsampling ratio• Costly pixel shaders:

SSAO, PCF soft shadow maps…

x2,2x2,2

x2,0x2,0

x1,9x1,9

x3,5x3,5

x3,0x3,0

x2,3x2,3

x3,7x3,7

x3,3x3,3

x2,4x2,4

40

Client-side applications

Client

3D stereo vision Temporal frameinter/extra-polation

No additionalbandwidth cost!

41

Video

42

Limitations

Inherited from upsampling method– multi-sample techniques are problematic: transparency, anti-aliasing, refractions,

reflections

Edge representation– more difficult to control bandwidth than with DCT-based codecs– requires more research for efficient RDO

High-res. ground truth image Upsampled image

43

Conclusions

Our scalable streaming framework– off-loads the server by moving part of the computation to

the client– auxiliary data stream allows for attractive applications on

the client side: 3D Stereo warping, temporal upsampling, in-game advertisement

– total bandwidth usage is comparable to streaming full-res. video using H.264

MC & DCT codecs are not an end-of-story for video compression!

44

Conclusions

Thank You for your attention!

45

Video

46

Backup: Custom encoder

Upsampling requires very precise values of depth and motion – especially at discontinuities

High quality also required for other applications (IBR) MPEG is not well suited for such data

– minimizes visual error globally– encoding hard edges requires lots of space with DCT

transform – encoding both depth and motion in one stream is

problematic (chroma subsampling, luma based MC)– solutions for some problems are defined but not

supported by many software/hardware vendors

H.264 encoder Our custom encoder

47

Temporally Interleaved Sampling

Temporal coherent sampling no new information for static frames Need disjoint sub-pixel sets for each frame

– We use a regular sampling pattern (efficient just jitter frames)

t

4n x 4n pixels

n x n pixels

48

Temporal Reprojection Caching

Naïve blending ghosting artifacts! Compensate for camera and scene motion, Easy!

?

t t+1

motion flow

warp disocclusions

49

Bibliography

[Herzog et al. 2010] - HERZOG R., EISEMANN E., MYSZKOWSKI K., SEIDEL H.-P.: Spatio-temporal upsampling on the GPU. In Proc. of I3D (2010), ACM, pp. 91–98.

[Didyk et al. 2010] - DIDYK P., EISEMANN E., RITSCHEL T., MYSZKOWSKI K., SEIDEL H.-P.: Perceptually-motivated real-time temporal upsampling of 3D content for high-refreshrate displays. Comp. Graph. Forum (Proc. of Eurographics) 29, 2 (2010), 713–722.

[Morvan et al. 2006] – MORVAN Y., H. N. DE WITH P., FARIN D.: Platelet-based coding of depth maps for the transmission of multi-view images. Proceedings of SPIE, Stereoscopic Displays and Applications (SD&A 2006)vol. 6055 p. 93--100, January 2006, San Jose (CA), USA

scalable remote rendering with augmented streaming

Documents

jittered framehighres

cloud computing

opengl es

scalable remote rendering

robert herzog

augmented streamingdawid

hanspeter seidel

karol myszkowski