Outatime: Using Speculation to Enable Low-Latency Continuous Interaction for Mobile Cloud GamingK Y U N G M I N L E E , D A V I D C H U , E D U A R D O Q U E R V O , J O H A N N E S KO P F, Y U R Y D E G T YA R E V, S E R G E Y G R I Z A N , A L E C W O L M A N , J A S O N F L I N N

U N I V E R S I T Y O F M I C H I G A N , M I C R O S O F T R E S E A R C H , S T. P E T E R S B E R G P O LY T E C H N I C U N I V E R S I T Y, S I B E R I A N F E D E R A L U N E V E R S I T Y

M O B I ’ S Y S ’ 1 5 , M AY 1 8 - 2 2 , 2 0 1 5

1

Background: Cloud GamingoProcessing and rendering done on cloud

oClient sends inputs, receives rendered images

oBenefitsoBetter graphics – use server’s processing hardware

oEasy to develop – no compatibility issues

2

The ProblemoLacks real-time interactivityoHigh latency sensitivity affects gameplay

oBuffering impossible due to changing user input

Fig. 1 (a) – Standard Cloud Gaming: Frame time depends on net latency

3

SolutionoSpeculate frames until next response

oChallenges – dynamism and sensitivity

Fig. 1 (b) – Outatime: Frame time is negligible

4

Outatime Architecture

5

Fig. 2: Outatime Architecture

Speculation for NavigationoCreate discrete time Markov ChainoDefine input 𝑁𝑡 = 𝛿𝑥,𝑡 , 𝛿𝑦,𝑡 , 𝛿𝑧,𝑡 , 𝜃𝑥,𝑡 , 𝜃𝑦,𝑡 , 𝜃𝑧,𝑡

oGiven input 𝑛𝑡, find most likely input for next frame 𝑁𝑡+1oFor RTT 𝜆:

𝑁𝑡+𝜆 = 𝑎𝑟𝑔𝑚𝑎𝑥 𝑝 𝑁𝑡+1 𝑁𝑡 = 𝑛𝑡 ∗ 𝑖=1𝜆−1 𝑝 𝑁𝑡+𝑖+1 𝑁𝑡+𝑖

oAlso track error estimate

6

Speculation for NavigationoSupersampling – Collect data as fast as input device allowsoImproves accuracy

oReduces sampling noise

oPrediction accuracy improves over 5 min.of samples

oUse training data of other playersoCharacteristics depend on skill level

7

Fig. 5 – Error distribution for different training periods

Speculation for NavigationoVideo ShakeoCaused by small prediction errors at low-latency

oFixed using Kalman Filtering

oKalman FilteroEmphasizes measured values for low RTT (< 40ms.)

oEmphasizes predicted values for high RTT (> 40ms.)

8

Misprediction CompensationoImage-based RenderingoTransform rendered prediction to be more accurate

oClipped Cube MapoRender areas surrounding frame in case they are needed

oLimit size based on expected error values

9

Clipped Cube Map

10

Fig. 7 – Cube Map Example

Image-based Rendering (IBR)

11

Figure 6 – Image-based Rendering in Fable 3

Speculation for Impulse EventsoMuch harder to predict

oSolution: speculate different possibilities in parallel

oCreate speculative input sequence

oAs RTT increases, speculative sequence space grows exponentially

oTwo methods to decrease speculative sequence sizeoSubsampling

oTime-Shifting

12

Subsampling and Time-ShiftingoSubsamplingoSample inputs at a period 𝜎 > 1 clock tick

oReduces state space to 2𝜆

𝜎

oOn its own, likely to miss samples

oTime-shiftingoShift every input activation to occur on the nearest subsample

oSolves problem of subsampling

oCan shift inputs backwards since state is speculative

13

14

Speculation for Impulse EventsoNot all inputs are binaryoAlternate firing for a weapon

oState space grows quickly (eg. 3𝜆 instead of 2𝜆)

oOutatime supports ternary and quaternary events for RTT ≤ 128ms

oSome impulse events delay tolerantoDo not speculate

oInstead, use time compression to account for RTT delay

15

Checkpoint and RollbackoSupports page-level and object-level checkpointingoDepends on density of Simulation State Objects (SSOs)

oPage-level checkpointingoCopy page on page write

oInvalidate mis-speculated data

oCopy back on rollback

oObject-level checkpointingoUse inverse functions when rolling back

16

ImplementationoManually modified Doom 3 code

oDoom 3 master with multiple speculative slave versionsorenderoundo

ocommit

orendercube

oUsed hardware to improve compression and video encoding

17

Experimento3 ExperimentsoDoom 3: 23 people

oDoom 3: 18 gamers

oFable 3: 23 people

oMeasured on 3 metrics:oMean Opinon Score

oSkill Impact

oTask Completion Time

18

ResultsoMinor decrease in quality for latencies up to 128 ms

oMore noticed by gamersoLarger/faster movements cause greater mispredictions

oMay be more sensitive as a player to such effects

oSignificant reduction in skills at higher latencies

oTask completion relatively unaffected

oImprovement over regular cloud gaming

19

Results

20

PerformanceoBandwidth 1.97x higher than standard cloud gamingo1.04 Mbps at RTT = 128ms

oFramerate = 52fps at 95th percentile

21

Strengths and WeaknessesoStrengthsoMany useful and practical tactics to minimize bandwidth

oEffective and varied predictive measures are taken for different classifications of inputs

oProvides foundation to make cloud gaming practical with relatively low latency

oWeaknessesoDoes not seem scalable to games with many inputs or fast inputs (eg. RTS)

oRequires significant code restructuringo Harder to use on existing games

o Does not consider faster framerateso Modern games are frequently 60Hz

22

Reference[1] Lee, Kyungmin, et al. Outatime: Using speculation to enable low-latency continuous interaction for mobile cloud gaming. Proc. of MobiSys. 2015.

23

Questions?

24

Image Source: http://i.ytimg.com/vi/gCSmykwODqA/maxresdefault.jpg

25

26