for mobile cloud gaming - semantic scholar · using graphics rendering contexts to enhance the...

Using Graphics Rendering Contexts to Enhance the Real-Time Video Coding

for Mobile Cloud Gaming

Shu Shi †, Cheng-Hsin Hsu ‡, Klara Nahrstedt †, Roy Campbell †

† Dept. of Computer Science University of Illinois at Urbana-Champaign, USA

ACM Multimedia, Scottsdale, Nov 29, 2011

‡ Deutsche Telekom Inc. R&D Lab, USA (Now at NTHU, Taiwan)

Tuesday, November 29, 11

What is Cloud Gaming?



http://www.wallcoo.net/game/car_race_series/html/wallpaper11.html





BroadbandInternet



BroadbandInternet

Mobile Cloud Gaming


Mobile Cloud Gaming

Rendering Results

RenderingServer

Mobile Client

Wireless Networks

User Interaction

RenderingViewpoint

Control Updates

3D Game Scene Data

Display


Mobile Cloud Gaming


Mobile Cloud Gaming

• Research Prototype

• Remote Rendering Games for iPhone

• BZFlag: an open source 3D tank shooting game


Mobile Cloud Gaming

• Research Prototype

• Remote Rendering Games for iPhone

• BZFlag: an open source 3D tank shooting game

• Demo Video


Benefits


Benefits

No Need of Powerful Graphics Hardware

Same Game Console 3D Experience

Cross-Platform Solution

Easy Distribution

Prevent Pirating

... ...


Challenges

Interaction Latency


Rendering Results

RenderingServer

Mobile Client

Wireless Networks

User Interaction

RenderingViewpoint

Control Updates

3D Game Scene Data

Display

Challenges

Interaction Latency


Challenges

Interaction Latency

• [Shi10] Shu Shi, Mahsa Kamali, Klara Nahrstedt, John Hart, Roy Campbell. A High-Quality Low-Delay Remote Rendering System for 3D Video. In ACM Multimedia (MM’10)

• [Shi09] Shu Shi, Won Jeon, Klara Nahrstedt, Roy Campbell. Real-Time Remote Rendering of 3D Video for Mobile Devices. In ACM Multimedia (MM’09)


Challenges

• Interaction Latency

• [Shi10] Shu Shi, Mahsa Kamali, Klara Nahrstedt, John Hart, Roy Campbell. A High-Quality Low-Delay Remote Rendering System for 3D Video. In ACM Multimedia (MM’10)

• [Shi09] Shu Shi, Won Jeon, Klara Nahrstedt, Roy Campbell. Real-Time Remote Rendering of 3D Video for Mobile Devices. In ACM Multimedia (MM’09)

Network Bandwidth


Network Bandwidth Usage

Mobile Devices

Video Requirements

Raw Bandwidth

320x240 @ 30fps 26.4 Mbps

640x480 @ 30fps 105 Mbps

1280x720 @ 30fps 316 Mbps


Real-Time Video Coding



• Low Coding Efficiency

• No bi-directional prediction

• Strict encoding deadline

• One-pass rate control

• Low buffer occupancy

• Intra refresh coding

• ... ...









• ... ...

• How bad is it?


0 10 20 30 40 50 60012345678910

Time (s)

Bit

Rat

e(M

bps)

DownlinkMean

OnLive Video Bandwidth Usage



0 1 2 3 4 5 6 6.505

10152025303540455055

Bit Rate (Mbps)Q

ualit

yin

PSN

R(d

B)

40 dB: Excellant Quality

x264 Encoding Performance

0 10 20 30 40 50 60012345678910

Time (s)

Bit

Rat

e(M

bps)

DownlinkMean



40dB: Excellent Quality


0 1 2 3 4 5 6 6.505

10152025303540455055

Bit Rate (Mbps)Q

ualit

yin

PSN

R(d

B)

40 dB: Excellant Quality

x264 Encoding Performance

0 10 20 30 40 50 60012345678910

Time (s)

Bit

Rat

e(M

bps)

DownlinkMean



6 - 7 times worse!

40dB: Excellent Quality









• ... ...


Video Coding Basics

I B B B P B B B P



1



1 2



1 2 3



41 2 3



541 2 3



6541 2 3



76541 2 3



876541 2 3



9876541 2 3



... ...9876541 2 3



... ...

• No B Frame!!!

PPPPPPI P P



P ... ...BBBPB1 B B



... ...1 BB B P BBB P



... ...1 BB BP BBBP


Auxiliary Frame Generation

... ...1 BB B BBBP P



... ...1 BB B BBB

P P



1



1 AUX1

1



1 AUX1 2

1 P

1

AUX1



31 AUX1 2

1

AUX1

1 P B



431 AUX1 2

1 P B

1

AUX1

B



5431 AUX1 2

1 P B B

1

AUX1

B



65431 AUX1 2

1 P B B B B

1

AUX1



AUX25431 AUX1 2

1 P B B B B



6AUX25431 AUX1 2

1 P B B B B P

AUX1

AUX2



76AUX25431 AUX1 2

1 P B B B B P B

AUX1

AUX2



76AUX25431 AUX1 2

1 P B B B B P B

AUX1

AUX2

8

B

... ...


Auxiliary Frame GenerationAUX AUX

1 P B B B B P B B B B


Basics of Video Coding

• Motion Estimation

http://www.cmlab.csie.ntu.edu.tw/cml/dsp/training/coding/motion/me1.html




Basics of Video Coding

• Search Method and Range

http://www.cs.cmu.edu/~yvchen/projects.html




3D Image Warping


3D Image Warping

• Input:

• Image: I1

• Depth: D1

• Viewpoints:

• v1

• v2

v1

v2


3D Image Warping

• Input:

• Image: I1

• Depth: D1

• Viewpoints:

• v1

• v2

• Output

• Image: I2

v1

v2


Double Warping


Double Warping

v1

v2


Double Warping

v1

v3

v2


Warping Based Real-Time Video Coding

I P B B B B P B B ... ...B B

AUX1AUX2


Warping Based Real-Time Video Coding

R R W W W W R W W ... ...W W

AUX1AUX2


Results

30

32

34

36

38

40

42

44

46

1 1.5 2 2.5 3

Qua

lity

in P

SNR

(dB

)

Bit Rate (Mbps)

Original x264 Encoding with only I/P framesProposed warping based encoding method using auxiliary frames

30

32

34

36

38

40

42

44

46

1 1.5 2 2.5 3Q

ualit

y in

PSN

R (

dB)

Bit Rate (Mbps)

A short sequence of 315 frames

A long sequence of 4072 frames


Discussion


Discussion

• Interaction Latency Reduction


Discussion


• Limitations

• Only applicable to cloud gaming or other remote rendering applications

• Mostly useful for first person perspective games

• Can’t handle graphical effects well


Discussion


• Limitations

• Only applicable to cloud gaming or other remote rendering applications

• Mostly useful for first person perspective games

• Can’t handle graphical effects well

• Optimizations

• Integration with x264


Conclusion

• Reduce video bandwidth for real-time coding in cloud gaming

• Generate auxiliary frames based on camera motion to enable B frame encoding

• Replace search based motion estimation with efficient 3D image warping using depth maps and camera vectors

• Think differently for different applications


Thank you!

• Acknowledgement

• The work is supported by Deutsche Telekom R&D Lab USA

• Contact Author

• Shu Shi: [email protected]

• Questions?


mailto:[email protected]

mailto:[email protected]

for mobile cloud gaming - semantic scholar · using graphics rendering contexts to enhance the...

Documents