Hitting 60Hz with the Unreal Hitting 60Hz with the Unreal Engine: Inside the Tech of Engine: Inside the Tech of

Mortal Kombat vs DC UniverseMortal Kombat vs DC Universe

Jon Greenberg – MK TeamJon Greenberg – MK TeamNathan Mefford – Chicago ATGNathan Mefford – Chicago ATG

Why Bother?Why Bother?

• In general, “twitch” games require very In general, “twitch” games require very high framerate.high framerate.

• Fast input response demands fast Fast input response demands fast feedback to playerfeedback to player

• Running at 60Hz a basic requirement of Running at 60Hz a basic requirement of fighting genre. fighting genre.

Why Is 60 So Rare?Why Is 60 So Rare?

• Very few games target 60Hz (< 10% of games)Very few games target 60Hz (< 10% of games)• Only 16.7 ms in which to do everything vs 33.3 Only 16.7 ms in which to do everything vs 33.3

ms at 30Hz. Implies half the time to do ms at 30Hz. Implies half the time to do everything… this is not correct.everything… this is not correct.

• In general, this means you have ~1/3 the time, In general, this means you have ~1/3 the time, due to fixed cost overhead which can’t be due to fixed cost overhead which can’t be removed.removed.

• Customer doesn’t “care” that you have less time Customer doesn’t “care” that you have less time to do everything – still wants game to look great.to do everything – still wants game to look great.

• Game must hit 60Hz on both PS3 and Xbox 360, Game must hit 60Hz on both PS3 and Xbox 360, and both versions look as close as possible!and both versions look as close as possible!

Why 1/3Why 1/3rdrd The Time? The Time?

• Game Game mustmust run at >= 60Hz – not allowed to run at >= 60Hz – not allowed to drop frames (bog).drop frames (bog).

• This means we have to set aside headroom that This means we have to set aside headroom that can absorb instantaneous spikes.can absorb instantaneous spikes.

• MK vs DC steady state ~= 9.5 ms per frame.MK vs DC steady state ~= 9.5 ms per frame.• Allows for lot of particle effects and variability. Allows for lot of particle effects and variability.

Other genres (even other fighting games) likely Other genres (even other fighting games) likely need a great deal less slack.need a great deal less slack.

• PhilosophyPhilosophy: Always address worst-case : Always address worst-case scenarios up front. scenarios up front.

The Problem (part 1)The Problem (part 1)

• Midway had decided to use UnrealEngine 3 Midway had decided to use UnrealEngine 3 (UE3) as basic middleware across all internal (UE3) as basic middleware across all internal games. Using UE3 was required by mgmt.games. Using UE3 was required by mgmt.

• UE3 was (is) designed for 30Hz FPS/3UE3 was (is) designed for 30Hz FPS/3rdrd person person action genre titles.action genre titles.

• We started with the October 2006 (post Gears of We started with the October 2006 (post Gears of War 1) codebase. Some additional features War 1) codebase. Some additional features taken from Epic taken from Epic àà-la-carte. Ex: MITV, file -la-carte. Ex: MITV, file caching, misc fixes.caching, misc fixes.

• About 22 months to develop the game.About 22 months to develop the game.

The Problem (part 2)The Problem (part 2)

• UE3 brings a lot to the table (nice tools, wide UE3 brings a lot to the table (nice tools, wide feature set) but imposes a lot of heavy feature set) but imposes a lot of heavy fixed fixed costscosts..

• There are also some choices made in the engine There are also some choices made in the engine that have problematic side effects for 60Hz play that have problematic side effects for 60Hz play (UObject overhead, Garbage Collection, etc).(UObject overhead, Garbage Collection, etc).

• Out-of-the-box fixed cost baseline (especially Out-of-the-box fixed cost baseline (especially GPU) too high for a 60Hz title. GPU) too high for a 60Hz title.

• Eg., Oct06 build GPU baseline ~ 9ms. Eg., Oct06 build GPU baseline ~ 9ms.

Breaking it DownBreaking it Down

• GPU OverheadGPU Overhead• GPU Fixed costsGPU Fixed costs• General rendering overheadGeneral rendering overhead• Multipass overheadMultipass overhead• Lighting costLighting cost• Particle costParticle cost

• CPU OverheadCPU Overhead• Particle costParticle cost• Cloth & WaterCloth & Water• Render thread virtual overhead/state cachingRender thread virtual overhead/state caching

GPU Fixed CostsGPU Fixed Costs

• Post-processingPost-processing• Usually the biggest fixed cost.Usually the biggest fixed cost.• Combine as many operations together as possible to Combine as many operations together as possible to

hide work (ie, Bloom+DOF+Gamma+Resolution hide work (ie, Bloom+DOF+Gamma+Resolution retarget)retarget)

• Cut as many corners as possible and special case as Cut as many corners as possible and special case as necessary – eg. we use 1 of 3 different DOF methods necessary – eg. we use 1 of 3 different DOF methods depending on the case:depending on the case:

• Normal gameplay: classic blur cross-fadeNormal gameplay: classic blur cross-fade• Main Menu/Cinematics: dialating Poisson discMain Menu/Cinematics: dialating Poisson disc• Klose-Kombat: a series of blur planes.Klose-Kombat: a series of blur planes.

• ““Normal” DOF+Bloom effect cost = 1.8 msNormal” DOF+Bloom effect cost = 1.8 ms

BloomBloom

• Bloom is done a little strangely to compensate Bloom is done a little strangely to compensate for linear color range and not having a separate for linear color range and not having a separate downsample/blur:downsample/blur:• Per environment thresholding value determines which Per environment thresholding value determines which

pixels bloom.pixels bloom.• Thresholding is done inside downsample pass and Thresholding is done inside downsample pass and

written out into the alpha channel as 0 or 1. written out into the alpha channel as 0 or 1. • This bloom mask is then blurred along with color.This bloom mask is then blurred along with color.

• We had separate thresholding and strength We had separate thresholding and strength values for characters and the general values for characters and the general background to allow the two to be tuned background to allow the two to be tuned differently. differently.

• Character masks were written/read from stencil Character masks were written/read from stencil buffer.buffer.

DistortionDistortion

• Normal UE3 distortion effect has 3ms overhead!Normal UE3 distortion effect has 3ms overhead!• Instead, fold Distortion into Translucency.Instead, fold Distortion into Translucency.• Sample from a snapshot of opaque pass, and do Sample from a snapshot of opaque pass, and do

a depth-based selection to prevent near-a depth-based selection to prevent near-distortion. distortion.

• Overhead now just “capturing snapshot” - just a Overhead now just “capturing snapshot” - just a copy blit of color buffer ~ 0.4ms.copy blit of color buffer ~ 0.4ms.

• Now usable everywhere!Now usable everywhere!• Optionally support recapture of the “snapshot” Optionally support recapture of the “snapshot”

per distorting effect to allow for layered distortion per distorting effect to allow for layered distortion effects as well. Needed for water level.effects as well. Needed for water level.

Motion BlurMotion Blur

• Very expensive to do full-screen. Very expensive to do full-screen. • Epic doesn’t support motion blurring of skinned Epic doesn’t support motion blurring of skinned

geometry! geometry! • Instead, motion blur effects done via rendering Instead, motion blur effects done via rendering

velocity-stretched fading geometry.velocity-stretched fading geometry.• Required changing GPU skinning (PC/360) and Required changing GPU skinning (PC/360) and

Edge (PS3-SPU) to support skinning against Edge (PS3-SPU) to support skinning against previous bone positions.previous bone positions.

• Requires localized blur-only Z-prepass to Requires localized blur-only Z-prepass to prevent additive blur effects from blending badly.prevent additive blur effects from blending badly.

Shadows and MSAAShadows and MSAA

• Game made use of MSAA-2x on both platformsGame made use of MSAA-2x on both platforms• Resolving MSAA is very expensive on PS3.Resolving MSAA is very expensive on PS3.• Combine full-screen modulated shadow blit with Combine full-screen modulated shadow blit with

MSAA color/depth resolve! MSAA color/depth resolve! • Hide heavy texture bandwidth operations inside Hide heavy texture bandwidth operations inside

math heavy shadow work. Shadow ALU math heavy shadow work. Shadow ALU overhead high enough that we can also hide the overhead high enough that we can also hide the Distortion copy blit!Distortion copy blit!

• No self-shadowing – disabled via stencil mask. No self-shadowing – disabled via stencil mask. • Once there’s no self-shadowing anyway, we use Once there’s no self-shadowing anyway, we use

proxy shadow characters.proxy shadow characters.• Total cost ~= 1.33msTotal cost ~= 1.33ms

FogFog

• Fullscreen per-pixel ~2 ms on the GPU.Fullscreen per-pixel ~2 ms on the GPU.• Visible vertices < visible pixels!Visible vertices < visible pixels!• Per-pixel fog is often overkill. Replaced with Per-pixel fog is often overkill. Replaced with

per-vertex fog and per-object fog (characters).per-vertex fog and per-object fog (characters).• To keep per-vertex costs low, only support 2 To keep per-vertex costs low, only support 2

active fog actors. active fog actors. • Heightfog is optional, and controlled via static Heightfog is optional, and controlled via static

branching.branching.• Also added optional undulating height fog, via Also added optional undulating height fog, via

pulsing sine-waves through the fog height.pulsing sine-waves through the fog height.• Dramatically cheaper!Dramatically cheaper!

General RenderingGeneral Rendering

• 8 bpc render targets, linear color scale of 0..2.8 bpc render targets, linear color scale of 0..2.• We light in a combination of We light in a combination of γγ=1.0=1.0 and and γγ=2.2=2.2, ,

depending on what we’re lighting, to save cost. depending on what we’re lighting, to save cost. • Opaque: uses MSAA Opaque: uses MSAA • Translucent: post-MSAA resolveTranslucent: post-MSAA resolve• Heavy use of Playstation Edge library for Heavy use of Playstation Edge library for

skinned and world geometry on PS3.skinned and world geometry on PS3.• 3D resolution of the game was 1040x624 which 3D resolution of the game was 1040x624 which

was then scaled up to allow the HUD to render was then scaled up to allow the HUD to render at 1280x720.at 1280x720.

Multipass OverheadMultipass Overhead

• Pass-per-light overhead is simply too high. Pass-per-light overhead is simply too high.

• We’re mostly prelit, so we chose forward We’re mostly prelit, so we chose forward rendering.rendering.

• Z-Prepass? Typical depth complexity < Z-Prepass? Typical depth complexity < 1.5.1.5.

• Loosely sort opaque objects front to back Loosely sort opaque objects front to back via “rings of detail”. Removing Z-prepass via “rings of detail”. Removing Z-prepass saves ~0.75 ms. saves ~0.75 ms.

• Touch each pixel only once if possible.Touch each pixel only once if possible.

World Lighting (static)World Lighting (static)

• World is prelit using Illuminate Labs’ World is prelit using Illuminate Labs’ Beast, with some “dynamic” RNMs built Beast, with some “dynamic” RNMs built with Turtle. Dynamic RNMs are with Turtle. Dynamic RNMs are animated in materials or via MITVs.animated in materials or via MITVs.

• Prelit lighting was a mix of texture and Prelit lighting was a mix of texture and vertex RNM lighting, with a fast-path vertex RNM lighting, with a fast-path added to support per-vertex diffuse only added to support per-vertex diffuse only RNM evaluation for distant objects.RNM evaluation for distant objects.

World Lighting (dynamic)World Lighting (dynamic)

• Effect point lighting is done via a mix of per-Effect point lighting is done via a mix of per-pixel lighting (floors) and per-vertex (the rest pixel lighting (floors) and per-vertex (the rest of the environment).of the environment).

• To account for maximum load, shaders are To account for maximum load, shaders are built with three diffuse-only point lights built with three diffuse-only point lights active and burned into the materialactive and burned into the material

• No branching! All three lights always No branching! All three lights always evaluated.evaluated.

• These lights are globally assigned and These lights are globally assigned and managed in 3-deep FIFO. managed in 3-deep FIFO.

Character Lighting (part 1)Character Lighting (part 1)

Custom lighting model:Custom lighting model:• Irradiance volume of SH coefficient sets.Irradiance volume of SH coefficient sets.• Eval gradients to determine an SH-set per Eval gradients to determine an SH-set per

object.object.• Diffuse light the model using only the first 4 Diffuse light the model using only the first 4

coefficients (“ambient” and “directional” term).coefficients (“ambient” and “directional” term).• The 3 effect point lights are evaluated per-vertex The 3 effect point lights are evaluated per-vertex

and combined into the final diffuse lighting result.and combined into the final diffuse lighting result.• Spec faked via power-scaling of (Spec faked via power-scaling of (EE••NN) and ) and

multiplying by diffuse lighting. multiplying by diffuse lighting.

SpecularitySpecularity

Character Lighting (part 2)Character Lighting (part 2)

• Skin transmission faked by using (Skin transmission faked by using (EE••NN) as lerp ) as lerp factor between diffuse lighting and SH ambient factor between diffuse lighting and SH ambient term. term.

• Rim Lighting: power-scaling (1-Rim Lighting: power-scaling (1-EE••NN) for falloff ) for falloff and then mul by hard thresholding (1-and then mul by hard thresholding (1-EE••NN). ).

• If threshold is raised high enough (~0.7), ends If threshold is raised high enough (~0.7), ends up looking like chrome mapping!. up looking like chrome mapping!.

• Final rendering cost ~= 0.8ms per characterFinal rendering cost ~= 0.8ms per character• Character mesh-chucks batch rendered.Character mesh-chucks batch rendered.

Skin and MetalSkin and Metal

The Story So Far…The Story So Far…

• So far costs are:So far costs are:• MiscMisc ~~0.5 ms0.5 ms• Shadowmaps: Shadowmaps: 0.5 ms 0.5 ms• Characters: Characters: 1.6 ms 1.6 ms• Environment: Environment: ~4.X ms~4.X ms• MSAA Resolve/Shadow:MSAA Resolve/Shadow: 1.3 ms 1.3 ms• PostFX: PostFX: 1.8 ms 1.8 ms• Total Total ~9.X ms~9.X ms

• What about particle effects?What about particle effects?

Particle EffectsParticle Effects

• Very large problem. Cascade not very Very large problem. Cascade not very optimal. optimal.

• Solution – port Cascade runtime async on Solution – port Cascade runtime async on separate worked threads (to SPU on PS3)! separate worked threads (to SPU on PS3)!

• All emitters for a particle system updated in All emitters for a particle system updated in single block of async work (particles, emitter single block of async work (particles, emitter state, system state). state, system state).

• All particle Modules ported to SPU, except All particle Modules ported to SPU, except for collision (due to data complexity). for collision (due to data complexity).

Particle Effects (CPU load)Particle Effects (CPU load)

• All per-particle overhead removed from All per-particle overhead removed from Game/Render thread! Game/Render thread!

• Particle overhead now a simple linear Particle overhead now a simple linear relationship between system count and relationship between system count and emitter count. emitter count.

• On PC/360, vertex data for sprites created On PC/360, vertex data for sprites created JIT by async worker thread. JIT by async worker thread.

• No changes/compromises to artist tools or No changes/compromises to artist tools or workflow.workflow.

Particle Effects (SPU load)Particle Effects (SPU load)

• SPUs extremely fast. SPUs extremely fast. • Just used basic C++ code (including Just used basic C++ code (including

templates and polymorphism). No need to templates and polymorphism). No need to bother with intrinsics or ASM. bother with intrinsics or ASM.

• Same module code runs on PS3/360.Same module code runs on PS3/360.• Complex (dependant) DMAs done Complex (dependant) DMAs done

synchronously. Simpler to deal with and synchronously. Simpler to deal with and fast enough that it doesn’t matter.fast enough that it doesn’t matter.

• Update done via SPURS jobUpdate done via SPURS job

Particle Effects (GPU load)Particle Effects (GPU load)

• GPU overhead less straightforwardGPU overhead less straightforward

• Attempt 1: Lie to hardware and tell it we’re Attempt 1: Lie to hardware and tell it we’re in MSAA-4x on non-MSAA target. Looks in MSAA-4x on non-MSAA target. Looks okay on wispy stuff in general (smoke, okay on wispy stuff in general (smoke, fire, etc.), but looks terrible on 360.fire, etc.), but looks terrible on 360.

Particle Effects (GPU cont…)Particle Effects (GPU cont…)

• Attempt 2: for somewhat opaque particles, break Attempt 2: for somewhat opaque particles, break effect out into masked pass and unmasked effect out into masked pass and unmasked pass, sorting particles for a system front to back pass, sorting particles for a system front to back before rendering to prime Z.before rendering to prime Z.

1.1.Render particles with alpha-test set to =1.0, front to backRender particles with alpha-test set to =1.0, front to back

2.2.Render particles with alpha-test set to <1.0, back to frontRender particles with alpha-test set to <1.0, back to front

• Didn’t help! Alpha-test disables ZCull writes, Didn’t help! Alpha-test disables ZCull writes, negating the benefits of the priming pass.negating the benefits of the priming pass.

• Attempt 3: Observation – for flipbook Attempt 3: Observation – for flipbook effects, lots of time is wasted effects, lots of time is wasted rendering alpha-0 space around rendering alpha-0 space around meaningful content.meaningful content.

• Idea: For flipbook effects, reduce Idea: For flipbook effects, reduce particle dimensions (and UVs) to particle dimensions (and UVs) to bound content of the particular bound content of the particular flipbook page!flipbook page!

• Works great! Dramatic fillrate Works great! Dramatic fillrate improvement from doing this (>50%). improvement from doing this (>50%). Requires artist to identify channel to Requires artist to identify channel to scan for image bounds.scan for image bounds.

Particle Effects (GPU cont…)Particle Effects (GPU cont…)

General Render Thread General Render Thread OptimizationsOptimizations

• Lots of work to reduce unnecessary operations.Lots of work to reduce unnecessary operations.• Render thread virtuals = death by a thousand Render thread virtuals = death by a thousand

paper cuts.paper cuts.• Cache as much state as possible to reduce Cache as much state as possible to reduce

redundant virtual calls. Eg, replaced redundant virtual calls. Eg, replaced FMaterialRenderProxy’s GetMaterial virtual call FMaterialRenderProxy’s GetMaterial virtual call with a caching call.with a caching call.

• Remove tons of unneeded repeated calls to Remove tons of unneeded repeated calls to GetXXX() (ie, GetPixelShader) states from GetXXX() (ie, GetPixelShader) states from inside Shader processing.inside Shader processing.

Misc Further optimizationsMisc Further optimizations

• Cloth simulation moved to run async in another Cloth simulation moved to run async in another thread (SPU on PS3).thread (SPU on PS3).

• Epic’s water simulation code ported to run on Epic’s water simulation code ported to run on SPU on PS3.SPU on PS3.

• Animation still synchronous Game-thread based, Animation still synchronous Game-thread based, but doesn’t use AnimTrees. Very limited blend but doesn’t use AnimTrees. Very limited blend options for designers.options for designers.

• No Occlusion pass – Vis is simple frustum No Occlusion pass – Vis is simple frustum culling.culling.

• Lots of work to reduce amount of memory Lots of work to reduce amount of memory allocation via pools and isolated heaps. Still, allocation via pools and isolated heaps. Still, accounts for 25% of CPU time.accounts for 25% of CPU time.

Garbage CollectionGarbage Collection

• Based on work by Stranglehold teamBased on work by Stranglehold team• Not quite as aggressive as they were, but Not quite as aggressive as they were, but

removes all live calling of GC from gameplay – removes all live calling of GC from gameplay – only called when exiting modes.only called when exiting modes.

• Memory management switched to deferred (by a Memory management switched to deferred (by a frame) cleanup of UObjects/AActors.frame) cleanup of UObjects/AActors.

• All “loaded” data trapped via RootsetAll “loaded” data trapped via Rootset• Introduces UResource class, a reference Introduces UResource class, a reference

counting UObject.counting UObject.• All USurface derived classes (ie, UMaterial, All USurface derived classes (ie, UMaterial,

UTexture, etc) are all reference counted via UTexture, etc) are all reference counted via UResource to prevent unwanted deletion. UResource to prevent unwanted deletion.

Additional Game DetailsAdditional Game Details

• We don’t use UnrealScript. Minimally use We don’t use UnrealScript. Minimally use Kismet. Use our own scripting engine (C/C++-Kismet. Use our own scripting engine (C/C++-ish) for AI, object management, menu logic, etc.ish) for AI, object management, menu logic, etc.

• Game scripts are expected to manage resource Game scripts are expected to manage resource lifetimes.lifetimes.

• Main advantage – dynamically reloadable for Main advantage – dynamically reloadable for fast iteration!fast iteration!

• MKScripts describe resource usage to determine MKScripts describe resource usage to determine cooked resources that need to be added to cooked resources that need to be added to characters/backgrounds.characters/backgrounds.

Artist LimitationsArtist Limitations

• UE3 gives artists a lot of rope to hang themselves UE3 gives artists a lot of rope to hang themselves with. with.

• Big thing was to limit who could use the Material Big thing was to limit who could use the Material Editor. Editor.

• All character art uses same small set of materials. All character art uses same small set of materials. • Characters budgeted at 20k polys visible at a time.Characters budgeted at 20k polys visible at a time.• Backgrounds budgeted based on visible object Backgrounds budgeted based on visible object

count and storage limitations more than polycount.count and storage limitations more than polycount.• Environment material/lighting complexity managed Environment material/lighting complexity managed

by the background lead to ensure overall by the background lead to ensure overall performance hit GPU performance targets, with performance hit GPU performance targets, with various metrics helping to tell them where they various metrics helping to tell them where they were.were.

General Recommendations for General Recommendations for hitting 60Hz in UE3hitting 60Hz in UE3

• Budget performance up front! Budget performance up front! • Given Edge and 360’s unified shaders, geometry Given Edge and 360’s unified shaders, geometry

less of a problem than fillrate. less of a problem than fillrate. • Predetermine valid PostFx and hardwire the Predetermine valid PostFx and hardwire the

majority of permutations. majority of permutations. • Reduce dynamic critical sectioned memory Reduce dynamic critical sectioned memory

allocation as much as possible. Massively stalls allocation as much as possible. Massively stalls all performance.all performance.

• Use pool allocators whenever possible, and Use pool allocators whenever possible, and watch for realloc’s.watch for realloc’s.

• Force designers and artists to run with Force designers and artists to run with performance metrics on!performance metrics on!

Recommendations for hitting 60Hz Recommendations for hitting 60Hz in UE3 on PS3 (well, and 360)in UE3 on PS3 (well, and 360)

• Consider what can be deferred and/or can be Consider what can be deferred and/or can be made to run async and consider moving that made to run async and consider moving that work.work.

• Consider using Edge on PS3. Consider using Edge on PS3. • Even sync’d work can be done way faster on Even sync’d work can be done way faster on

SPU if divided over multiple SPUs/threads! SPU if divided over multiple SPUs/threads! • Don’t be intimidated by the SPUs on PS3. Don’t be intimidated by the SPUs on PS3.

Prototype SPU code on 360/PC where its Prototype SPU code on 360/PC where its easier to debug.easier to debug.

• Template heavy C++ might not be ideal Template heavy C++ might not be ideal performance case for SPUs, but certainly a performance case for SPUs, but certainly a LOT better than not using them at all. LOT better than not using them at all.

Things We Have Yet to AddressThings We Have Yet to Address

• Serialization – as we tend to only stream Serialization – as we tend to only stream content underneath movie playback or content underneath movie playback or load screens, the CPU impact wasn’t too load screens, the CPU impact wasn’t too problematic for us, though it does impact problematic for us, though it does impact load times.load times.

• Animation – need to explore making it run Animation – need to explore making it run on worker threads/SPU for deferrable on worker threads/SPU for deferrable (background and LOD’d) objects.(background and LOD’d) objects.

Questions?Questions?

• Thanks for listening!Thanks for listening!