designing tools enhance interactive experiences & the development process

Designing Tools Designing Tools EnhanceEnhance

Interactive Interactive ExperiencesExperiences

& the development & the development processprocess

Magy Seif El-NasrMagy Seif El-NasrCollege of ISTCollege of IST

Penn State UniversityPenn State University

Interactive Entertainment: Interactive Entertainment: Industry ImpactIndustry Impact

Interactive Entertainment: Interactive Entertainment: Industry ImpactIndustry Impact

Interactive Entertainment: Interactive Entertainment: Industry ImpactIndustry Impact

Game industry profits: 2002: $1 billion more than movie industry 2004: EA reported $1.43 billion quarterly earning 2005: Nintendo $4 billion and expects to sell at

least 2 million Wiis in US alone

Importance of Game Engines + toolsImportance of Game Engines + tools

Entertainment Training

Leaders project, ICTDevil May Cry IIEducation

Virtual Aquarium, NCSU

code

Tools

.radix 16 ;all numbers hexadecimal

start: mov #41,@#8264 ;system variable loop: jsr r4,print1 .asciz "HELLO WORLD! "

br loop print1:jsr pc,@#1248 ;print string (ROM call)

rts r4

System.Writeline (“Hello World”);

Assembly

C#

Learning and Computer literacy ImpactLearning and Computer literacy Impact

Several Researchers believe thatSeveral Researchers believe that

“video games provide an easy lead-in to computer literacy” – Cassell 2000

High Schools students and Modding

Books on Modding for teens gaining popularity

Interactive Entertainment:Interactive Entertainment: Motivation and LearningMotivation and Learning

“I want to learn how to make an awesome video game. I would like to learn everything about technology or at least more than I did”

- middle school girl

“I want to make my characters talk, build a world, and make interesting stories”

- another middle school girl

Impact Impact and and Importance Importance ofof the toolsthe tools

Facilitate development: games, training, health therapy application

Facilitate learning: media, computing literacy

Facilitate Creativity

Two thrusts of my researchTwo thrusts of my research

Tools for building engaging interactive

experiences

Game Engines + Tools as learning

environments

Tools for building engaging interactive

experiences

Game Engines + Tools as learning

environments

Two thrusts of my researchTwo thrusts of my research

Focus: visual design and character design

Based on cinematic and theatric theory

Using CI and AI techniques

My ResearchMy Research

Tools Lighting Camera (in progress) Character (in progress) Dance (in Progress)

Why Lighting?Why Lighting?

Lighting is Everything Lighting is Everything

MiiMii

• BS in Computer Science BS in Computer Science

• Directed in TheatreDirected in Theatre

• Graphics Design in AdvertisementGraphics Design in Advertisement

• MS in Computer Science MS in Computer Science

• Studied Psychology and NeuroscienceStudied Psychology and Neuroscience

• PhD in Computer SciencePhD in Computer Science

• 2 years studied acting & lighting design at 2 years studied acting & lighting design at Northwestern Northwestern

Why Lighting?Why Lighting?

Dramatic Tension

Visual Focus

Style

Feel of the space

Time of Day

Period

visibility

Lighting in Traditional MediaLighting in Traditional Media

Manually setting light layout Manually setting light layout (light maps)(light maps)

Advantages:Advantages: realisticrealistic Controllable Controllable

Disadvantages:Disadvantages: Don’t adapt to variations Don’t adapt to variations

in the environmentin the environment Requires much memoryRequires much memory

Game Lighting Game Lighting Static Lighting Design Static Lighting Design

Images from Lightmaps (static shadowmaps) article written by Kurt Miller from:http://www.flipcode.com/articles/article_lightmaps.shtml

x =

Image from Max Payne

Manually scripting Manually scripting Lighting effectsLighting effects

Dynamic Character Dynamic Character LightingLighting

Real-time ShadowsReal-time Shadows

Dynamic user control of Dynamic user control of some lightssome lights

Game Lighting Game Lighting Static + Dynamic LightingStatic + Dynamic Lighting

Game Lighting Game Lighting Static + Dynamic LightingStatic + Dynamic Lighting

Game Lighting Game Lighting Static + Dynamic LightingStatic + Dynamic Lighting

Advantages:Advantages:More realisticMore realisticAdaptable to changeAdaptable to change

Disadvantages:Disadvantages:Effects are scripted and rely on very careful Effects are scripted and rely on very careful designdesignRestricted Restricted Based on realismBased on realismCompositional Balancing is done at design timeCompositional Balancing is done at design time

Problems: No modulationProblems: No modulation

Screenshot from Mission 21 Devil May Cry

Saturation Graph for Screenshot

Problems (Blade of Darkness): No Problems (Blade of Darkness): No adaptation to game Playadaptation to game Play

Problem 1: unpredictabilityProblem 1: unpredictability

Develop a lighting plot or setup based on:Develop a lighting plot or setup based on:Narrative configuration:Narrative configuration: StoryStory Conflict/dramatic tensionConflict/dramatic tension

Timing and dramatic progressionTiming and dramatic progressionPhysical Configuration:Physical Configuration: Camera orientation Camera orientation

and positionand position Characters positions Characters positions

and orientations and orientations

Requiring dynamic adaptationRequiring dynamic adaptation

depend depend on useron user}

Two GoalsTwo Goals

Dynamic Adaptive Lighting:Dynamic Adaptive Lighting:

Better gameplayBetter gameplay

Facilitate new Interactive modelsFacilitate new Interactive models

High-Level authoring of lighting:High-Level authoring of lighting:

Faster prototyping Faster prototyping

Exploration of design spaceExploration of design space

Developing such as a Developing such as a DesignDesign Tool is Hard Tool is Hard

Technical Issues of Technical Issues of DynamicDynamic Lighting Lighting Artistic Artistic control control for stylefor style low-level balancing of low-level balancing of contextcontext and and lighting parameterslighting parameters but provide but provide continuitycontinuity

Why?

ELE – ELE – Expressive Lighting EngineExpressive Lighting Engine

A lighting system that:A lighting system that:

IntelligentlyIntelligently adjusts adjusts lighting in real-time accommodatelighting in real-time accommodate contextcontext and and effecteffect

Based on Based on cinematic cinematic & theatric & theatric theorytheory

Allow Allow artistartist to to controlcontrol lighting at a lighting at a high-levelhigh-level

ELEELE

The three subsystems:The three subsystems: use optimization to find best solution use optimization to find best solution given context, desired effects, state, and artists’ constraintsgiven context, desired effects, state, and artists’ constraints

Game/Rendering Engine

ELE

Allocation Subsystem

Angle Subsystem

Color Subsystem

Current State

Previous State

LAMP (Lighting Action Message Protocol)

WAMP (World Action Message Protocol)

Artistic Constraints

ELEELE

The three subsystems:The three subsystems: use optimization to find best solution use optimization to find best solution given context, desired effects, state, and artists’ constraintsgiven context, desired effects, state, and artists’ constraints

Game/Rendering Engine

ELE

Allocation Subsystem

Angle Subsystem

Color Subsystem

Current State

Previous State

LAMP (Lighting Action Message Protocol)

WAMP (World Action Message Protocol)

Artistic Constraints

Automatic light allocationAutomatic light allocation

find best allocation find best allocation given given artistic artistic constraintsconstraints:: ModelingModeling Depth Depth VisibilityVisibility Visual ContinuityVisual Continuity Visual FocusVisual Focus Low vs. high keyLow vs. high key

Algorithm for dynamic allocationAlgorithm for dynamic allocation

1.1. Calculate visible area Calculate visible area

2.2. Divide the zone into overlapping areasDivide the zone into overlapping areas

3.3. Allocate a number of lights to areas,Allocate a number of lights to areas,given that given that minimize: minimize:

visibility modeling Visual

continuitydepth

:p L A

arg max( ( ) ( ) ( ) ( ))

opt

v d m vcp

p

V p D p M p VC p

Layout of lightsLayout of lights

ELEELE

The three subsystems:The three subsystems: use optimization to find best solution use optimization to find best solution given context, desired effects, state, and artists’ constraintsgiven context, desired effects, state, and artists’ constraints

Game/Rendering Engine

ELE

Allocation Subsystem

Angle Subsystem

Color Subsystem

Current State

Previous State

LAMP (Lighting Action Message Protocol)

WAMP (World Action Message Protocol)

Artistic Constraints

Lighting angle SelectionLighting angle Selection

Select azimuth, elevation angles, Select azimuth, elevation angles, given given artistic constraintsartistic constraints : :

Visual ContinuityVisual Continuity Motivation of directionMotivation of direction VisibilityVisibility ModelingModeling MoodMood

ELE finds best light angles to meet goalsELE finds best light angles to meet goals

Automating cinematic lighting designAutomating cinematic lighting designChoosing key light azimuth angleChoosing key light azimuth angle

Optimizes:Optimizes:

• kk is the key light angle, is the key light angle, kk-- is angle previous frame is angle previous frame• mm is the mood angle desired, V is visibility is the mood angle desired, V is visibility vv is the cost of deviation from best visibility angle is the cost of deviation from best visibility angle -- is the cost of visual continuity is the cost of visual continuity ll is the cost of deviation from realistic direction is the cost of deviation from realistic direction mm is the cost of deviation from ideal mood angle is the cost of deviation from ideal mood angle

cost( , , , )

(1 ( , )) minv m l ii

k s k m

V k s k k k m k l

visibility & modeling

visual continuity

mood motivation

Evaluation of key light angle Evaluation of key light angle visibility & modelingvisibility & modeling

( , ) sin( )cos( )V k s k sWe described these rules as :

Nose line

Light

Camera

s

k

From: Millerson’s

The technique of lighting for television and Film, 1991

Artist's setting

0

20

40

60

80

100

120

Co

sts

ELE: Angle SubsystemELE: Angle Subsystem

Showing character’s facial Showing character’s facial expressions and gesturesexpressions and gestures

Artist's settings

0

20

40

60

80

100

120

Co

sts

Showing character’s facial Showing character’s facial expressions and gestures, high expressions and gestures, high

tensiontension

ELE: Angle SubsystemELE: Angle Subsystem

Mood Angle = side angle

Artist's settings

0

20

40

60

80

100

120

Co

sts

Emphasizing mood and Emphasizing mood and mysterymystery

ELE: Angle SubsystemELE: Angle Subsystem

Mood Angle = side angle

Artist's settings

0

20

40

60

80

100

120

Co

sts

Emphasizing moodEmphasizing mood

ELE: Angle SubsystemELE: Angle Subsystem

ELEELE

The three subsystems:The three subsystems: use optimization to find best solution use optimization to find best solution given context, desired effects, state, and artists’ constraintsgiven context, desired effects, state, and artists’ constraints

Game/Rendering Engine

ELE

Allocation Subsystem

Angle Subsystem

Color Subsystem

Current State

Previous State

LAMP (Lighting Action Message Protocol)

WAMP (World Action Message Protocol)

Artistic Constraints

Choosing ColorsChoosing Colors

Color (Hue + saturation Color (Hue + saturation

+ intensity)+ intensity)

Compose colors for Compose colors for different areas on the setdifferent areas on the set

Choosing ColorsChoosing Colors

Adjust colors to accommodate desired Adjust colors to accommodate desired artistic artistic constraintsconstraints:: Depth Depth Dramatic IntensityDramatic Intensity Dramatic focusDramatic focus low vs. high key settinglow vs. high key setting Specific author-suggestedSpecific author-suggested

Hue, Saturation, Lightness, color Warmth for Hue, Saturation, Lightness, color Warmth for focus, non-focus, and backgroundfocus, non-focus, and background

Palette restrictions specifying stylePalette restrictions specifying style

+ maintain + maintain visual continuityvisual continuity and and stylestyle

Choosing ColorsChoosing Colors

Optimize:Optimize:

depth

21

2

1

{ , , }

{ , , }

( , ) ( )

contrast ( )

( )

( , )

( ),

t t td

tc

t

t tvc i i

i f n b

ti

i f n b

cost c c D c d

c

p c

E c c

I c

contrast

Palette constraints

Visual continuity

Artist’s desired colorparameters

Choosing ColorsChoosing Colors

2

2

2

2

( )

( )

( )

( )

i

i

i

i

ts i i

th i i

tl i i

tw i i

S c s

H c h

L c l

W c w

( )tiI c

Saturation

Hue

Lightness

Warmth

Calculating Color WarmthCalculating Color Warmth

Based on warmth perceptionBased on warmth perception

Used linear fit to psychophysical data:Used linear fit to psychophysical data:

Warm

Cool0.008

0.0006 0.422

0.0105

TR R

warmth G G

B B

Artist's setting

0102030405060708090

Idea

l V

alu

es

warm tones, warm tones, low color contrastlow color contrast

for low tension, realistic colorsfor low tension, realistic colors

Lightness (NF)Contrast (F – NF)Warmth (NF)

ELE: Color Subsystem ELE: Color Subsystem

cool tones, cool tones, low color contrastlow color contrast

for night scenes,for night scenes,or emphasize characteror emphasize character

Artist's setting

010203040

50607080

Idea

l V

alu

es

ELE: Color Subsystem ELE: Color Subsystem

Artist's setting

0102030405060708090

100

Idea

l V

alu

es

warm, warm, high intensity contrasthigh intensity contrast

for high tensions scenesfor high tensions scenes

ELE: Color Subsystem ELE: Color Subsystem

Two GoalsTwo Goals

Dynamic Adaptive Lighting:Dynamic Adaptive Lighting:

Better gameplayBetter gameplay

Facilitate new Interactive modelsFacilitate new Interactive models

High-Level authoring of lighting:High-Level authoring of lighting:

Faster prototyping Faster prototyping

Exploration of design spaceExploration of design space

Dynamic Intelligent Dynamic Intelligent Cinematic Lighting Cinematic Lighting

can make a difference in gamescan make a difference in games

Use Light as tension meterUse Light as tension meter

Demo (Dynamic Lighting)Demo (Dynamic Lighting)

Demo (Static Lighting)Demo (Static Lighting)

Results of Pilot Study – Results of Pilot Study – during during CHI 2005CHI 2005

Non-players liked the visual representation Non-players liked the visual representation

Liked representation of information through Liked representation of information through lightinglighting

Some players said: it made the game too easySome players said: it made the game too easy

Some gamers said: lighting was disturbingSome gamers said: lighting was disturbing

Adaptive Visual Focus Adaptive Visual Focus

can make a difference in gamescan make a difference in games

Adapting Difficulty of Spotting Adapting Difficulty of Spotting enemiesenemies

Two GoalsTwo Goals

Dynamic Adaptive Lighting:Dynamic Adaptive Lighting:

Better gameplayBetter gameplay

Facilitate new Interactive modelsFacilitate new Interactive models

High-Level authoring of lighting:High-Level authoring of lighting:

Faster prototyping Faster prototyping

Exploration of design spaceExploration of design space

ElevatorElevator

ElevatorElevator

EDISON Edge Detection and Image EDISON Edge Detection and Image SegmentatiON SegmentatiON [Christoudias et al. [Christoudias et al. 2002]2002]

Mean-shift approach Mean-shift approach [Comaniciu and [Comaniciu and Meer 2002]Meer 2002]

Design Galleries Design Galleries [Marks et al. 1997][Marks et al. 1997]

User adjusting Lighting in Games like User adjusting Lighting in Games like The The Sims Sims or or The MoviesThe Movies

A comment from an educator:“This can be used as a way of promotingMedia Literacy”

Classes – Using the ToolsClasses – Using the Tools

400-level interdisciplinary course on Interactive Narrative, tools: C# Flash ELE Interactive Narrative

Engine

Classes – Using the toolsClasses – Using the tools

400-level Game Design, tools:

Unreal + ELE Half Life 2 Torque

Classes – Exploring the theoryClasses – Exploring the theory

400-level Interdisciplinary course on Design of Immersive Environments

Future DirectionsFuture Directions

More on lightingMore on lighting

Lighting: industry partnersLighting: industry partners

Continue on Projects:Continue on Projects: CameraCamera CharacterCharacter DanceDance

Tools for building engaging interactive

experiences

Current ProjectsCurrent Projects

Ambient Intelligence and Dancedesigning tools to allow artists to dictate aesthetic effect

Pressure + Physiological

Sensors

Intelligent Systems

Current ProjectsCurrent Projects

Virtual Actor Training

Comedia dell’arte in Second Life

Character, gesture, and camera tools

Current ProjectsCurrent Projects

Lockheed Martin Lightingvisual designcamera character

Living Hypothesis

Future DirectionsFuture Directions

Use more of the research Use more of the research toolstools

Impact of the use of the Impact of the use of the research toolsresearch tools

Game Engines + Tools as learning

environments

Specifically our work on Games and Modding

Classes – Using Game ModdingClasses – Using Game Modding

Middle/high school Gaming for Girls

Future Work: Camera tool Lighting Character

In ConclusionIn Conclusion

ConclusionConclusionGame Industry is a growing industryGame Industry is a growing industry

Game tools are importantGame tools are important

In my research I develop new tools toIn my research I develop new tools to Enhance design processEnhance design process Enhance interactive experienceEnhance interactive experience

Application of these tools: Education, Application of these tools: Education, Training, and other applicationsTraining, and other applications

Leaders project, ICTDevil May Cry II

Questions?Questions?

http://faculty.ist.psu.edu/http://faculty.ist.psu.edu/seifel-Nasr/seifel-Nasr/

magy@ist.psu.edumagy@ist.psu.edu

Modding and LearningModding and Learning

TransferTransfer of Computer Science, Math, Art of Computer Science, Math, Art ConceptsConcepts

Different Different engines promote engines promote different different concepts, concepts, and require different and require different pre-reqspre-reqs

Increase self efficacyIncrease self efficacy of high-school female of high-school female studentsstudents

MaterializeMaterialize abstract concepts abstract concepts

Learn something about Learn something about themselvesthemselves

Different engines are Different engines are popular popular among different among different groupsgroups

Concepts Promoted thru ModdingConcepts Promoted thru Modding

Essential ConceptsEssential Concepts AssumptionsAssumptions

of engines need to of engines need to be understood, e.g.be understood, e.g.Parallel processing in Parallel processing in Warcraft IIIWarcraft III

Concepts you need know to Concepts you need know to operateoperate the the engine, engine,

Co-ordinate systemCo-ordinate systemColor, contrast, etc. to use ELEColor, contrast, etc. to use ELE

Assignment-dependant conceptsAssignment-dependant concepts, e.g., e.g. Character ModelingCharacter Modeling

Modding and LearningModding and Learning

TransferTransfer of Computer Science, Math, Art of Computer Science, Math, Art ConceptsConcepts

Different Different engines promote engines promote different different concepts, concepts, and require different and require different pre-reqspre-reqs

GraphicsGraphics: Vector Math, Geometry, Animation: Vector Math, Geometry, Animation ProgrammingProgramming: Parallel processing, event : Parallel processing, event

programming, Object Oriented Programmingprogramming, Object Oriented Programming ArtsArts: Architecture Design, map design, visual : Architecture Design, map design, visual

Composition, Lighting, camera movement, etc.Composition, Lighting, camera movement, etc.

Game Projects involvedGame Projects involved

Project ManagementProject ManagementGroup WorkGroup Workiteration designiteration designPrototypingPrototypingCritiqueCritique

Game Design concepts: Game Design concepts: MechanicsMechanics RulesRules etcetc

What do students learn when they What do students learn when they are building games using the are building games using the

current tools?current tools?