The Tech Behind the Tools of Insomniac Games

Geoff Evans – Insomniac Games

Welcome!

• An overview of how Insomniac’s tools get current generation games out the door

• There are no slick demos of our game engine

• This talk is focused tools internals, pipelines, and best practices we have found

Outline

Areas of improvement:• Revision control• Asset organization• Flexible Asset definition• Efficient Asset Processing

Warning: Open Source Ahead

• Nocturnal Initiative is Insomniac's Open Source project for tools code & more

• Portions available for download under BSD-style license, long term initiative

• Wiki:– http://nocturnal.insomniacgames.com

• Perforce:– nocturnal.insomniacgames.com:1666

Problem

• Proprietary revision control system• Implicit file retrieval– Files synced before open– Per-file database overhead– Written files queued for check-in at process exit– Heavy code instrumentation (iopen, iclose)

• Damage spreads immediately• All processed data under revision control

Solution

• Move all revision control to Perforce– Handles branching of binaries efficiently– Label code and data together

• Explicit asset retrieval– Users decide when to download– Big speed improvement– Better stability while getting a unit of work done

• Remove built data from revision control

Branching

• We branch all files for milestones• Work continues in the main line during

milestone stabilization• Milestone fixes brought back piecemeal• Milestone hacks stay in branch forever

Continuous Integration

• Custom system implemented in Perl• Populates a ‘Known Good’ label in Perforce• Failure notifies users based on recent

checkins (changes between testing and last known good changelist numbers)

• Moving forward we intend to sync to ‘Known Good’ by default (code and assets)

Problem

• Problem: Asset tree organized by engine type• Tools and engine expected numbered assets– Moby 500: Wooden Breakable Crate– Level 9000: My Test Level

• Storage location of each asset was fixed or had a natural home in the tree:– /Data/Mobys/Moby500/actor.xml (fixed location)– /Art/Mobys/Moby500/crate.mb (natural home)

Problem

• Numbered assets are simple to program for• Reference any asset using type and id• Lacked descriptive names, hard to find,

hindered re-use of assets• Users tracked assets with pen and paper• Changing the type of an asset lead to complex

copy logic that had to copy files around

Solution

• Tools shouldn’t force asset organization• Users should be able to put files anywhere• Assets are defined in files with a reasonable

names (BreakableCrate.entity vs. moby500)• Files should be able to be moved or renamed

without a lot of work• Renaming a single file is simple, but references

to that file are troublesome

Asset Referencing

• Historically references are file path strings• Makes renaming content files difficult and

error prone– Asset A references B, renaming B to C breaks A

• Housecleaning before a sequel game causes pain due to broken references

• Typically fixed up by brute force

Asset Referencing

• Don’t rely on string paths for referencing files• Use a unique IDs to indirect the file location• IDs are assigned and shared via events• Event files are retrieved with asset files• Event files contain simple event logic:– File <foo> was added with id <blah>– File <foo> was moved to <bar>– File <bar> was deleted

Event System

• Each event is written to a file per-user, per-computer, per-project, per-branch– “geoff-geoff_xp-rcf-devel.event.dat”– Avoid contention for file checkout– Avoid conflict resolution during integrate

• EventSystem collates events, sorts by time• Events handled only once

File Resolver

• File Resolver maps unique ID -> File path• Consumes events from Event System• Populates a local machine SQLite DB• Native formats store only the unique ID• Maya stores ID in dynamic attributes– String paths are fixed up at load time by plug-in

• TUID – 64 bit unique ID– Open Source via Nocturnal

Asset Referencing

• Processed data is output to folders named for the file id of the Asset definition file

• Even though assets are renamed, processed data location remains the same

• Helps efficiency on the back end of pipeline• It’s possible to reference assets in game by the

file ID by hand, but we try to avoid this

Here be Dragons

• Since databases are built on each computer, it can have tricky machine-specific bugs (revert)

• We commit Add and Rename operations immediately to fight missing asset references (dangling TUIDs), data files may be valid stubs

• Moving forward we think its possible to store resolver data as meta-data and eliminate collation of Events altogether

Problem

• Problem: Assets defined by engine type specific data structures– Our assets were defined in terms of our engine,

not in terms of the content our engine was consuming

– This made changing the engine type arduous since the data could be very different

Solution

• Asset data should be modular, and that modularity should drive the engine type– Asset definition should serve the content, not

specific engine types (wherever possible)– Similar data about Assets should be reusable

between engine types

Asset Fundamentals

• What is an Asset: A discrete unit of content that needs to be processed by one or more builders to be loaded by the game

• An Asset itself has only basic properties• An Asset is a collection of Attributes– Asset : public AttributeCollection

Assets (C++ Classes)

Attributes

• Attribute: A block of related information– AnimationAttribute: what animations to use– PhysicsAttribute: physics subsystem info– VisualAttribute: render information

• Attributes store the bulk of the complexity• Stored in ‘slots’ using class type id (int32)

Attributes (C++ Classes)

Asset Example

• EntityAsset – content backed placeable object– ArtFileAttribute – reference Maya content file– VisualAttribute – render information

Asset Example

• LevelAsset – top level chunk of game data– WorldFileAttribute – placed objects in the level– DependenciesAttribute – list of assets to pack

Asset Example

• ShaderAsset – texture and render properties– ColorMapAttribute – texture, compress settings– NormalMapAttribute – processed differently

Attributes

• Simplifies sharing structures between Assets• Attributes only exist for features that are

configured by content creators (auditing)• Easy to copy and paste between Assets

Validating Attributes

• If we aren’t careful users could easily create non-sensical Assets– LevelAsset with a TextureMapAttribute?– Shader with a PhysicsAttribute?

• Must be able to validate Attributes– Attribute may not work with some Asset classes– Attributes may not work in conjunction

Attribute Behavior

• Behavior defines default compatibility• Exclusive: ColorMapAttribute– Validated by ShaderAsset, not by LevelAsset

• Inclusive: DependenciesAttribute– References to miscellaneous resources– Useful for spawn-only types like weapons, HUD– Loosely related assets like weapons and ammo

Attribute Usage

• Usage implies Class/Instance relationship– Class – Used in classes only– Instance – Used in instances only– Overridable – Works in both class and instance

• Allows us to easily override class settings per instance on an Attribute level

Attribute Usage

• Our SceneNode class also subclasses AttributeCollection:

Attribute Usage

• BakedLightingAttribute defines light map dimension and compression settings– Instances of that EntityAsset can override the light

map information (if its scaled way up or down)

Attribute Sibling Validation

• Some Attributes inherently conflict• Attributes can reject potentially invalid siblings– Ex: BakedLightingAttribute will reject the addition

of PhysicsAttribute because baked lighting doesn’t work with dynamic objects

Engine Type Classification

• Assets map to one or more Engine Types• If Asset map to one type, it’s a natural match– ShaderAsset -> ShaderBuilder– LevelAsset -> LevelBuilder

• If Asset class maps to more than one– Attributes determine Engine Type– EntityAsset -> ShrubBuilder, TieBuilder, etc…

Entity Assets

• Shrub – light weight static art– ArtFileAttribute – Maya content file– VisualAttribute – render information

Shrub

Entity Assets

• Tie – more expensive static art– CollisionAttribute – makes it collidable– BakedLightingAttribute – lightmap size

Tie

Entity Assets

• Moby – animated character– PhysicsAttribute – Physics subsystem settings– AnimationAttribute – Animation assets to use

Moby

Too many Attributes!

• Plethora of Attributes is confusing to artists• Keeping Attributes simple is key• Attributes are cumbersome because knowing

which Attributes yield each Engine Type its costly to learn

Just make it a <Engine Type>!

• Sometimes users know what Engine Type they want, and need the ability to conform the Attributes to that Engine Type

• Attributes have Enable flag which can be used to disable offending Attributes so data is not lost, just dormant

• Changing the type back re-enables the disabled Attributes

What Attributes do I need again?

• Setting up Assets from scratch can still be non-trivial… basic Attributes are needed to get an Asset up and running

• We created a Wizard with simple presets for each Engine type

• Moving forward we are going to select creation preset based on export data contents

Problem

• Problem: .NET was not very productive for generating UI for editing our Assets– We don’t want to give up writing Native code,

so .NET in general was not a good fit for us– Marshalling data structures between Native and

Managed was time consuming– Windows::Forms::PropertyGrid was hard to deal

with to get good custom property editing

Solution

• Drop .NET and develop our own solutions– Our C++ reflection was getting feature-rich– We already had a data-driven properties GUI– Switched to C++ / wxWidgets– Back up and running after 1 man month– Simplified everything

Asset Editor UI

• Assets and Attributes are loaded into a single tree view– References to other Assets are expanded in-place– Properties displayed for each node in the tree– Property interaction can update information in

real-time on the target platform

Asset Editor UI

What’s under the hood

• We generate property GUI for Asset/Attributes from C++ reflection information

• Adding a field to an Asset or Attribute shows up automatically

• UI-script can add sliders, drop-downs, color pickers, etc…

Reflect

• Provides class and field level meta-data• All meta-data created at runtime– No compile time parsing required

• Adding a field is usually a single line of code• Type Registry assigns type id’s at runtime• Automated persistence, cloning, visitor API• Handles data format changes & legacy files• Open Source via Nocturnal

Reflect Classesclass Version : public Reflect::ConcreteInheritor<Version, Element>{public: std::string m_Source; static void EnumerateClass( Reflect::Compositor<Version>& comp ) { comp.AddField( &Version::m_Source, "m_Source" ); }};

void Init(){ Reflect::RegisterClass<Version>( "Version" );}

Reflect Enumerationsnamespace BlendTypes{ enum BlendType { Normal, Additive };

static void EnumerateBlendType ( Reflect::Enumeration* info ) { info->AddElement(Normal, "Normal"); info->AddElement(Additive, "Additive"); }} typedef BlendTypes::BlendType BlendType;

void Init(){ Reflect::RegisterEnumeration<BlendType>( "BlendType”,

&BlendTypes::EnumerateBlendType );}

Inspect

• System for data-driven property GUIs• Control class hierarchy– Wrapper classes for wxWidgets UI elements– Can be factory created from UI script

• Interpreter class hierarchy– Intimate with data formats– Automates the creation of the UI– Handles callbacks from user interactions

• Open Source via Nocturnal

ReflectInspect

• Interpreter classes that generate Inspect UI• Binds control state to in memory structures• Basic support for STL containers + Math types• Support for custom UI handling via plug-ins• Easy to integrate custom property elements• Open Source via Nocturnal

Holy crap that’s a lot of Properties!

• This system is awesome for drowning our users in lots of properties

• Agility is good, not the most important thing• Cull properties that are used very infrequently• Adding support for filtering properties• Adding support for categorizing properties• Adding support for dependencies to hide

other properties when not applicable

Problem

• Users spend a lot of time waiting for Assets to build– No longer getting built data from revision control– Building major animated characters took 15+ min– Building levels took 30+ min

Solution

• Build system must share data between users• Information about an output file is hashed

into a single signature for the output:– Hash of input file paths, format version strings– Hash of input file contents or folder contents– Hash of dependency relationships– Hash of output file paths, format version strings

Cache Files

• This signature used to build a file path to a output cache file on the network

• That file is downloaded instead of built• Locally built data is uploaded automatically

when the build phase completes• Downloading updates the modification time of

the cache file on the network– Script trims down cache based modification time

Format Version

• Format version is a string compiled into tools• Changed a programmer changes a data format– “15 – geoff: improved shader flossiness”

• Very efficient because users only reprocess data when data formats or assets change

• Takes discipline when checking in changes to libraries that affect lots of output formats

Build Process

1. Given list of assets to build2. Create builder objects for assets3. Builders compute output signatures4. Attempt download of output files from cache5. Build what could not be downloaded6. Upload built files to cache

Cache Files

• Started with Distributed Cache System (DCS)• Query for files on other workstations around

the office, central server for tracking files• DCS was complex, wasn’t well suited• Moved to a Central Cache System (CCS)• Now we have files sharing throughput issues

with lots of concurrent users• Windows 2008 / Linux are possible solutions

Cache Files

• Cache file storage is just an optimization• Any built data can be regenerated at any point• VPN users have separate cache for their site if

they have multiple machines• Continuous iteration system is fantastic for

seeding build cache files

Summary

Major wins for our tools:• Perforce• File Resolver• Assets & Attributes• Built File Cache

Thanks!

geoff@insomniacgames.com