Saints Row Scheduler

Randall Turner, Volition, Inc. Randall.turner@volition-inc.com Discussion Areas:

Saints Row project Saint’s Row scheduler

Saint’s Row

Single Thread, CPU Bound

Alpha Kits

Two Hardware Threads Dual Core

Hardware thread Software thread Many software threads on one hardware

thread Software threads != concurrency. Hardware

threads do.

First Step: Alpha kit solution Sim/Render Split (Still CPU bound,

not as bad.) Two HW Threads

Beta Kits

Betas = Six hardware threads, Woo Hoo! But each hardware thread was slower than Alphas. Homogeneous processing, unified memory

First Beta Soluton

Problem

Still only using mostly two HW threads (Sim, Render)

Badly CPU bound No more major “natural” separation

points

Interim Solution

Manually move processing to other HW threads.

Example – Shadow Calculations Create a SW thread to process half of

the object list Process half the list when triggered.

Fixed Concurrent Processing

Uses an entire SW thread for one type of processing.

Wastes system resources Hard coded to execute that processing type only. Inflexible, Scales up badly.

Solution… Scheduler & Jobs

Goal Make all movable so will use

available space

Scheduler (Job Manager)

Jobs Worker threads Job Queues Triggers

Jobs

Independently schedulable entity No dependencies on other jobs. Atomic with respect to other jobs Essentially a function/data pair

Creating Jobs

Extract parallelism from loops. (Splatting)

Segregate memory usage Thread safe interactions with systems

critical sections double-buffering

Examples

Compute shadow silhouettes Character animation Object visibility culling Building command buffers Bad candidate - AI processing

Not easily turned into jobs due to interaction with large, difficult to thread safe data set.

Basic Scheduler Flow

Worker Threads

One per HW thread Draws jobs from job queues Associated with job “type”

Sim jobs, Render jobs, or “any” job. “Soft” or “Hard” linkage Dynamically configurable

Wakes up when jobs go into job queue Can be deactivated

Job Queues

One per job type Sim or Render

Implicit prioritization High frequency, contention sensitive

Used critical sections on version 1.0

Triggers Fire on completion of jobs Scheduling Triggers – schedule more

jobs Event Triggers - activate suspended

threads Can tie together to create state

machines.

First Pass

Final Pass 90% CPU Utilization

Flow-Around Callback to scheduler on interrupt

Allow “small” jobs to complete, then resume Jobs contain duration info

Havok

Timestep called from Scheduler job More flexible

Problem: Has own internal scheduler Timestep Goes “Serial” Jobs don’t flow

Solution: break out and splat serial processing ourselves.

Havok - serial extraction

Performance Critical Sections

Scheduler v1.0 all critical sections Blockage risk – CS owner preemption

Can stall ALL threads

Lock-Free Faster – x2 no contention, x10-20 high-

contention situations NO blockage risk (Woo Hoo!)

GP Gems 6, Fober

Job Size

Scheduler overhead determines job size. Small as possible. Assume 5% overhead Saints Row “ideal” size about 250

microseconds 7 microseconds per job, 5% = 150 microsec, 250 drops to ~ 3% overhed

Questions?

For latest slides: http://www.volition-inc.com/gdc07/

Extras Pix with PVS on, night, high load

Release, w/o PIX samples, ~ 35ms

Lock Free URL’s

Fober implementation: http://www.grame.fr/pub/LockFree.pdf

http://www.grame.fr/pub/TR-050523.pdf

Toby Jones: GP Gems 6 Good coding style