LuxembourgJanuary 28, 2013

European Commission Information Society and

Media

2nd Review Meeting

GaLAGame and Learning Alliance

The European Network of Excellence on Serious Games

WP3 – SG Application Fields

Michela Mortara, Chiara Eva Catalano, Bianca Falcidieno

CNR – IMATIMarco Luccini CEDEP

LuxembourgJanuary 28, 2013

European Commission Information Society and

Media

2nd Review Meeting

Outline

• Overview of WP3– mission and objectives

• Synthesis of Year 1 – activities and outcomes

• Year 2– Focus, strategy, outcomes

– Plan for user studies

• Plan for year 3 (4)

LuxembourgJanuary 28, 2013

European Commission Information Society and

Media

2nd Review Meeting

Overview of WP3

• Mission: Understand the current practices, needs and desiderata of customers and developers of SGs in different application domains

• 6 specific Tasks, or Special Interest Groups

– Business and Management (T3.1, CEDEP)– Engineering and Manufacturing (T3.2, POLIMI)– Health and Fitness (T3.3, HWU)– Security, Safety & Crisis Management (T3.4, NURC)– Humanities & Heritage (T3.5, CNR)– Personal & Social Learning & Ethics (T3.6, INESC-ID)

• 1 Harmonization & Integration Task (T3.7, CEDEP)

LuxembourgJanuary 28, 2013

European Commission Information Society and

Media

2nd Review Meeting

Synthesis of Year 1

Activity: surveying…

• SG proposition and diffusion dynamics• Dissemination channels• Key stakeholders• Barriers to the adoption • Requirements• Open issues

Outcomes:

• Collection of SGs in the application domains• Best practices in adoption/development• Recommendations for maximizing SG diffusion

STAR

questionnaires, interviews

LuxembourgJanuary 28, 2013

European Commission Information Society and

Media

2nd Review Meeting

Main findings of year 1

• High demand of evidence based practice

• Lack of a formal methodology to assessgame effectiveness perceived as the mainbarrier to SG adoption

• Scepticism about SG effectiveness

• Low adoption, small market

• Uncertain return of investment

• Few skilled developers, low budget projects

Scepticism about SG

effectiveness

Low adoption,

small market

Uncertain return of

investment

Few developers, low budget

projects

LuxembourgJanuary 28, 2013

European Commission Information Society and

Media

2nd Review Meeting

Focus of Year 2

“The panel recommends that this WP develop a plan to design and conduct one or more empirical studies of SGs now in use which are generally regarded as effective but which lack empirical evidence of impact. Such a research plan could be developed in year two and conducted in year three”

Based on the 1st year outcomes and the reviewers’ recommendation, the focus of year 2 has been on the learning impact of SGs.

LuxembourgJanuary 28, 2013

European Commission Information Society and

Media

2nd Review Meeting

Strategy

SG descrip

tion

AS – game

testing and evaluation

London – game

workshops

SG collection (year 1) SG monitoring (year 2)

• Game workshops

• Game testing and evaluation

• Collaborative SG descriptions

• Educational objectives & exploitation styles

• Technological features

• Availability to test

• Success– Diffusion / Popularity

– Awards

• Learning Impact– Evidence from the literature

– Recognised potential

LuxembourgJanuary 28, 2013

European Commission Information Society and

Media

2nd Review Meeting

Strategy

SG descrip

tion

AS – game

testing and evaluation

London – game

workshops

SG collection (year 1) SG monitoring (year 2)

• Game workshops

• Game evaluation and testing

• Collaborative SG descriptions

For each SIG:

• Best practices wrt maximizing the learning impact

• Current/best practices in the assessment methodology

• Identification of the SGs for user studies

LuxembourgJanuary 28, 2013

European Commission Information Society and

Media

2nd Review Meeting

BPs and recommendations on how to maximize and evaluate the Learning Impact [T3.7]

• Minimum cognitive overload

• Facilitate the learning

Minimizing the cognitive overload

• R1• Engage the learner

experientially/constructively• Situate the learning

Maximizing the learning experience

• Clear Evaluation strategies

• Comprehensive evaluation plan

Designing learning evaluation as an Impact project

• R3

• Flexibility, reusability, exploitation

• Facilitate the learning

Design the SG as an integral part of a learning program

R1 R2

R3 R4

LuxembourgJanuary 28, 2013

European Commission Information Society and

Media

2nd Review Meeting

BPs and recommendations on how to maximize and evaluate the Learning Impact [T3.7]

• Minimum cognitive overload

• Facilitate the learning

Minimizing the cognitive overload

R1

• User friendly game interface & Intuitive navigation (Playing History, Plantville, Siqur)

• No previous knowledge required, gradually increasing difficult learning tasks (Shortfall, Thiatro)

• Use facilitation when the embedded complexity is relevant

LuxembourgJanuary 28, 2013

European Commission Information Society and

Media

2nd Review Meeting

BPs and recommendations on how to maximize and evaluate the Learning Impact [T3.7]

• R1• Situate the learning• Engage the learner

experientially/constructively

Maximizing the learning experience

R2

• Minimization of cognitive overload a necessary condition• A suitable environment and proper interactions to fit the context of use

at best• Immersive 3D environments (Thiatro, Boarders’ Ahoy!, TARGET,

Innov8)• Interaction among players (EagleRacing)

• Engage learners in building their new knowledge (Icura, MeTycoon)• Fun but also emotional tension and frustration

LuxembourgJanuary 28, 2013

European Commission Information Society and

Media

2nd Review Meeting

BPs and recommendations on how to maximize and evaluate the Learning Impact [T3.7]

• Clear Evaluation strategies

• Comprehensive evaluation plan

Designing learning evaluation as an Impact project

• R3

• Facilitate the learning

Design the SG as an integral part of a learning program

R3 R4

• Identify as early as possible the learning goals and the kinds of data to be collected in order to identify the suitable learning metrics.

• If the game is part of an educational strategy the stakeholders will be involved for the design and evaluation phase

• Facilitation to raise attention, provide learning guidance and consolidate the learning

• Maximization of the learning impact and effective evaluation

LuxembourgJanuary 28, 2013

European Commission Information Society and

Media

2nd Review Meeting

Plan for user studies – objectives

Learning theories and

models

best practice

s & recommendation

s

Plan for user

studies• Selected SGs generally regarded as

effective

• Go beyond available studies (if any)

• Validate the GALA evaluation (description template -WP1)

• Evaluate SG effectiveness per se / with respect to or coupled with other means

• Assess long term retention / actionability

• Not only cognitive gain, higher levels in Bloom’s / Kirkpatrick’s models

LuxembourgJanuary 28, 2013

European Commission Information Society and

Media

2nd Review Meeting

User studies

• EagleRacing (collaboration dynamics)• MetaVals (finance)• Seconds (supply chain management)• Cosiga (product development process)• SBCE (lean manufacturing)• Re-Mission (support to treatment)• Squire’s Quest (food and nutrition)• Afghanistan pre-deployment course (defence, cultural

awareness) • Boarders Ahoy! (defence, ship boarding operations)• Playing History – The Plague (History)• ICURA (cultural awareness, japanese etiquette)• Stop Disaster! (risk awareness)• Darfur is Dying (human rights)

LuxembourgJanuary 28, 2013

European Commission Information Society and

Media

2nd Review Meeting

User studies

• EagleRacing (collaboration dynamics)• MetaVals (finance)• Seconds (supply chain management)• Cosiga (product development process)• SBCE (lean manufacturing)• Re-Mission (support to treatment)• Squire’s Quest (food and nutrition)• Afghanistan pre-deployment course (defence, cultural

awareness) • Boarders Ahoy! (defence, ship boarding operations)• Playing History – The Plague (History)• ICURA (cultural awareness, japanese etiquette)• Stop Disaster! (risk awareness)• Darfur is Dying (human rights)

Is part of some company specific programs and other education programs at CEDEP and at

INSEAD

The evaluation will benefit of a PGLC, never done before

LuxembourgJanuary 28, 2013

European Commission Information Society and

Media

2nd Review Meeting

User studies

• EagleRacing (collaboration dynamics)• MetaVals (finance)• Seconds (supply chain management)• Cosiga (product development process)• SBCE (lean manufacturing)• Re-Mission (support to treatment)• Squire’s Quest (food and nutrition)• Afghanistan pre-deployment course (defence, cultural

awareness) • Boarders Ahoy! (defence, ship boarding operations)• Playing History – The Plague (History)• ICURA (cultural awareness, japanese etiquette)• Stop Disaster! (risk awareness)• Darfur is Dying (human rights)

Developed and adopted at POLIMI

Summative evaluation including scenario questions

LuxembourgJanuary 28, 2013

European Commission Information Society and

Media

2nd Review Meeting

User studies

• EagleRacing (collaboration dynamics)• MetaVals (finance)• Seconds (supply chain management)• Cosiga (product development process)• SBCE (lean manufacturing)• Re-Mission (support to treatment)• Squire’s Quest (food and nutrition)• Afghanistan pre-deployment course (defence, cultural

awareness) • Boarders Ahoy! (defence, ship boarding operations)• Playing History – The Plague (History)• ICURA (cultural awareness, japanese etiquette)• Stop Disaster! (risk awareness)• Darfur is Dying (human rights)

Largely adopted and well studied

BUT will be evaluated from the point of view of game mechanics and narrative

LuxembourgJanuary 28, 2013

European Commission Information Society and

Media

2nd Review Meeting

User studies

• EagleRacing (collaboration dynamics)• MetaVals (finance)• Seconds (supply chain management)• Cosiga (product development process)• SBCE (lean manufacturing)• Re-Mission (support to treatment)• Squire’s Quest (food and nutrition)• Afghanistan pre-deployment course (defence, cultural

awareness) • Boarders Ahoy! (defence, ship boarding operations)• Playing History – The Plague (History)• ICURA (cultural awareness, japanese etiquette)• Stop Disaster! (risk awareness)• Darfur is Dying (human rights)

Is part of education programs at CEDEP and at INSEAD and at some company specific programs.The evaluation will benefit of a PGLC, never done

before

Both considered BP for CA in defence operations – different technological solutions

Planned a comparative study on different user profiles

LuxembourgJanuary 28, 2013

European Commission Information Society and

Media

2nd Review Meeting

User studies

• EagleRacing (collaboration dynamics)• MetaVals (finance)• Seconds (supply chain management)• Cosiga (product development process)• SBCE (lean manufacturing)• Re-Mission (support to treatment)• Squire’s Quest (food and nutrition)• Afghanistan pre-deployment course (defence, cultural

awareness) • Boarders Ahoy! (defence, ship boarding operations)• Playing History – The Plague (History)• ICURA (cultural awareness, japanese etiquette)• Stop Disaster! (risk awareness)• Darfur is Dying (human rights)

Largely adopted in Danish schools.

Evaluation at DIS (Genova) in march, to be repeated over time

LuxembourgJanuary 28, 2013

European Commission Information Society and

Media

2nd Review Meeting

User studies

• EagleRacing (collaboration dynamics)• MetaVals (finance)• Seconds (supply chain management)• Cosiga (product development process)• SBCE (lean manufacturing)• Re-Mission (support to treatment)• Squire’s Quest (food and nutrition)• Afghanistan pre-deployment course (defence, cultural

awareness) • Boarders Ahoy! (defence, ship boarding operations)• Playing History – The Plague (History)• ICURA (cultural awareness, japanese etiquette)• Stop Disaster! (risk awareness)• Darfur is Dying (human rights)

Large diffusion (supported by U.N.)

positive perception of learning impact but no empirical studies yet

Evaluation of experimental vs control group

LuxembourgJanuary 28, 2013

European Commission Information Society and

Media

2nd Review Meeting

Further SIG activities (WP3 Handbook)

• Set-up and maintain dedicated SIG web pages (as suggested by the reviewers) [joint effort with WP8]

• Contribute to SG description

• Publish scientific papers [16 ref. proc, 2 ref. journals] , organize Special Issues [1] and Workshops [3]

• Attract Associate Partners

• Monitoring SG proposition, stakeholders, dissemination channels

LuxembourgJanuary 28, 2013

European Commission Information Society and

Media

2nd Review Meeting

Next steps

• Main efforts in year 3: – Conduct the user studies– Disseminate the outcomes– Recommendations for maximizing the total value of SGs:

revision, update, validation through the user studies – Continue Evaluating/describing SGs, Monitoring SG proposition,

Updating the SIG pages

• Outlook to year 4:– describing and prioritizing the kinds of research that is needed to

measure the learning impact (reviewers’ rec.) from the application perspective

– SIG Final Thematic Workshops

LuxembourgJanuary 28, 2013

European Commission Information Society and

Media

2nd Review Meeting

Thank you for your attention