copyright replay solutions, llc 2004 game|tech 2004 building a reusable replay system jonathan lindo...

Copyright Replay Solutions, LLC 2004

game|tech 2004

Building A Reusable Replay System

Jonathan LindoJeff Daudel


Introduction

Jonathan Lindo & Jeff Daudel Company founders Research, next-gen technology at Nortel

Networks, Lucas Arts, Secret Level, and SGI.

Replay Solutions, LLC Founded in 2003 In use with Ion Storm, Crystal Dynamics,

NVIDIA Corp.


Why Have A Recording System?

Why do I need one? Great uses for Recording Systems.

Nailing down bugs Communicating bugs from QA to Dev Memory & Performance Profiling Demos


What is a Recording System? VCR Functionality

Record Mode Captures data that will be necessary to execute

the program accurately at a later point in time. Replay Mode

Intercepts non-deterministic calls. Injects previously recorded data into the game.

What type of data should replay record?


Sources of Non-Determinism

User input Timers Asynchronous callbacks Thread context switching DMA access timing Shared memory access Interrupts


Types of Recording Systems

A brief overview of recording systems:

External I/O Recorders. Source Code Instrumenters Binary Patching Systems


External I/O Recorders

Network IO, User Input, D3D Recorders

Pros: Don't require any mods to game code. Are robust to code and asset changes. Can be used for cross platform testing, demos.

Cons: May not produce accurate replays due to Timing issues,

race conditions. May be rejected by system for security reasons (anti-

replay attacks on the xbox)


Source Code Instrumenters

Proxy API Libraries, Modules

Pros: Somewhat Reusable Can be easily expanded, tuned. Recordings may be portable across platforms.

Cons: Applicable only to modules for which source is available. Requires the use of specialized API for certain calls by the

developer, or a code parsing module. No support for 3rd party modules. May not produce 100% accurate replays.


Binary Patching Systems

Post-compilation Binary Instrumentation

Pros: Very Reusable Deep level recording will reproduce accurate recordings. No source code mods required, can be applied to any EXE, LIB or DLL. Zero impact when disabled. Can capture low-level ASM calls.

Cons: Can be fragile when code or assets change. May require special support for certain APIs. Support required for processors and binary formats (PE, XBE, ELF) Difficult to make recordings portable between platforms. Cross module inlining (i.e. LTCG) distorts function boundaries


Recording System Architecture

Game Binary

Game CodeStatic Libs, DLLs

Kernel / Operating System

Processor(s), System Hardware

Recording System

Network I/O

File I/O

Multi-Threading

User Input

Interrupts

Timers


Replaying Async Callbacks

Define & Install Proxy callback handlers Record & Queue events in Proxy handlers Deliver callback events at pre-determined

Sync points, such as Proxy methods. During Replay, deliver simulated callback

events at recorded Sync points. This methodology can be applied to Windows

message callbacks, sound system callbacks, or async I/O of almost any kind.


Replaying Async Callbacks

Game Binary

Kernel / Operating System

Processor(s), System Hardware

Recording System

Proxy Callback Handlers

Game Callback Handlers


Replaying Thread Context Switches

Thread context switching order is mostly important when data is being exchanged between threads.

Identify the methods which are called to share data between threads. These are Thread Sync Points. Lock, TryLock, Release, EventWait, EventSignal WaitForMultipleEvents, WaitForSingleObject EnterCriticalSection, LeaveCriticalSection

By ensuring that these methods are called in the same order, we can make sure that data is processed in the same way on replay as it was on record.


Replaying Thread Context Switches

Recording System

Thread A

Thread B

Shared Resource


Robustness and Portability

Handling hardware configuration differences when replaying on different machines.

Syncing assets and source code in a dynamic development environment.


Handling Hardware Differences

To support replaying recordings on machines with different configs, steps must be taken during recording to ensure replay compatibility.

Determine a baseline hardware config that will be supported.

Create Proxy functions for all hardware detection methods. (GPU, Sound Hardware)

Force the game to the baseline hardware config to ensure compatibility when replaying on different systems.*

*(Make this an optional setting!)


Syncing Assets & Code

In a live dev environment, code and assets change daily. This makes it difficult to store valid Recordings.

When recording, save accessed assets and code relating to the build for future replays. This can be done through File I/O Proxy functions.

When replaying, load assets and code relating to the recording for reference.

This can be implemented using a simple storage and retrieval module, local or network based.


Feature Additions

Fast forwarding… Any function that does not affect game state can

be skipped during Replay.

Skipping ahead to a saved game state. Grab game & system state Provide an API interface to dump and restore

game state.


Memory

Accurate memory replication is required to: Reproduce Memory leaks Reproduce any code that has data sorted

by pointers (i.e. stl collection of pointers)

Typically, engines main loop involves iterating over game objects that are sorted by their memory address.


Memory Isolation Provide a private heap for all game memory.

Intercept all memory requests to the OS and redirect to a controlled heap. HeapAlloc, VirtualAlloc, etc.

Isolates nondeterministic system calls from making under-the-hood allocations.

Track system memory allocations requested from the game. Console environments may ask the game to provide and

manage memory (XMemAlloc on xbox) Simulate these requests on replay

Dummy memory, but will insure game heap is preserved.


Replay Performance

Importance of low impact Can be run at all times – not just when looking

for a bug Toughest bugs can occur at unexpected times Does not have to be removed from shipping

version. Captures more realistic game scenario Accessible throughout the entire QA process

Especially near the end where it is needed most!


Being Efficient Record only what is nondeterministic.

Most function calls are deterministic in nature – no need to record them rand()

Optimize high-performance clock sampling using sample throttling. Besides high performance profiling, usually there

is no need to access time more than a few times a frame.


Buffering Buffer replay data and flush to persistent

storage only when needed. Disk and network storage I/O is expensive. Flush periodically or when a critical event

occurs Exceptions, exit

Install exception handlers and filters to intercept these events.


Practical Results Applied a replay binary replay patcher on

several titles Typical replay data can be as low as 300

KB / minute Will vary for each game Depends on game’s functionality

No noticeable drop in frame rate.


Shared Memory Access

Difficult to record & replay! Enforce the use of a Recording

System API when accessing shared memory from Game Code. Allows replay system to monitor shared

memory

copyright replay solutions, llc 2004 game|tech 2004 building a reusable replay system jonathan lindo...

Documents

copyright replay solutions

xbox slide

types of recording systems

source code mods

reusable deep level

modules pros

accurate recordings

proxy handlers