Using Games for Assessment: Why? How? And ExamplesJenn McNamara

BreakAway Games

Serious Play 2016 Chapel Hill, NC

Before we start we must remember… Games aren’t real – images from “If games were real” cracked.com

Topics

■ A little bit about assessment ■ A little bit about games’ qualities to support

assessment ■ Examples demonstrating their intersection

GamesAssessment

Assessments Serve To…

■ Measure outcomes or knowledge levels Attainment of specific levels of proficiency

■ Provide insight into the learning / thought processes■ Provide immediate feedback

Specific gaps Potential areas or direction for further instruction

■ Provide a record of behavior■ Certify proficiency

Types of Assessment

■ Formative Assessment: formal and informal assessment procedures employed by teachers during the learning process in order to modify teaching and learning activities to improve student attainment. Typically qualitative feedback that focuses on the details of content and performance.

■ Summative Assessment: seeks to monitor educational outcomes, often for purposes of external accountability.

■ Standardized Assessment: questions, conditions for administering, scoring procedures, and interpretations are consistent and are administered and scored in a predetermined, standard manner.

Wikipedia

Engagement

Fundamental components of the gameplay experience: Analysing immersion Ermi & Mayra

Support for Games for Learning

■ Well designed games can enhance motivation and learning a sense of accomplishment: Bandura; Gist et al. informative feedback: Bransford et al.; Salas & Cannon Bowers challenge /competition: Epstein & Harackiewicz; Reeve & Deci

■ Provide successive tasks to help players make progress towards concrete, specific and timely goals goal setting: Locke et al.; Locke & Latham; Schunk & Ertmer

■ Provide repeated interactive experiences in task based environment expertise: Glaser; Chi et al.; Bransford et al anchored instruction: Bransford et al.; CGTV active learning: Rothman; Chi; Mayer; Vogel et al.

Entertainment Software Association 2012; Hussain et al 2009

Games and Formative vs. Summative vs. Standardized Assessment

■ Games play nicely for formative assessment and learning Fundamental teaching element in support of formative assessment

they provide constant, clear feedback that allows for correction and learning

■ Relationship between standardized or summative assessment and games is potentially problematic

■ An ideal standardized or summative assessment purposely avoids teaching in the interest of maintaining validity – consider simulation vs. game or redefine assessment

9

USC Standard Patient Studio

In the US, adequate diagnostic interview & physical exam are the #1 and #2 sources of indemnity payout.

PIAA Data Sharing ProjectDSP2009-2013; 2014Edition.

Desire standardized, repeatable, virtual, emotionally expressive, standard patient encounters for medical diagnostic skills and interview training without expenses of live actors and recorded video evaluation.

Funded by the Defense Medical Research Development Program.

USC Standard Patient Studio -Simulated patient interactions with performance measurement / formative guidance. -Allows spoken or typed interviews, branching encounters, diagnostic tests and assessment.- Reduced authoring from 6 months to < 1 day with no coding.

VALIDATION- >68% case performance gains after 20 minutes of use- 92% recognition & 96% accuracy medical interview natural language - automatic medical performance assessment accurate within 5%RECOGNITION 2015 Gold Serious Play Award 2015 IMSH SGVE Challenge Winner

10

Standardized Patient Studio Feedback

11

Standardized Patient Studio Feedback

12

TraumaSim / RAMPART

University of Pittsburgh physician / researcher studying trauma referral behaviors of ED physicians couldn’t simulate real world performance in the laboratory using traditional written cases.

Pitt researchers validated that using TraumaSim: 1. physicians made decisions

consistent with actual practice 2. that cognitive load could be

manipulated3. load increased use of

heuristics consistent with cognitive theory

RECOGNITION2015 Serious Play Bronze Award

Mohan, D., et al. Assessing the Validity of Using Serious Game Technology to Analyze Physician Decision Making (2014) http://dx.doi.org/10.1371/journal.pone.0105445

Created a simple 2D environment:- present multiple patients - manage common distractionswith complimentary goals:- study physician decision making- evaluate construct validity of

using a serious game for this purpose

13

TraumaSim / RAMPART

14

TraumaSim / RAMPART

15

The NBCOT Navigator™

INSTITUTE OF MEDICINE REPORTConcept for new tools derived from IOM Report: Health Professions Education: A Bridge to Quality

PRACTICE ANALYSIS STUDYinput from 2,500 certificants asking for interactive experiencesVALIDATED 6 DOMAINS•Client-Centered Care•Work in Interprofessional Teams•Utilize Informatics•Employ Evidence-Based Practice•Apply Quality Improvement•Professional Responsibility

VIRTUAL CONTINUING COMPETENCY ASSESSMENT PLATFORM Created with BreakAway Games to offer accessible, engaging and dynamic tools to provide:• Feedback on current skills• Career enhancement / growth• Evidence-based resources

PILOT TESTED AND LAUNCHEDTotal # of users: 512 & 13,227Total # of tools : 6,561 & 31,141 (9 week pilot & first 38 weeks live)

RECOGNITIONJudge’s Choice Most Innovative Product Demo - Association of Test Publishers Conference 2016Silver Serious Play Award 2016

16

NBCOT Navigator™: A Portal + 4 Types of Assessments

PORTAL AND CERTIFICANT DASHBOARD

SELF REFLECTIVE ASSESSMENT

AND MINI PRACTICE

QUIZES

ANIMATED CASE SIMULATIONS

GAMES

Contact Information

Jenn McNamaraVice President Serious Games and Strategic Partnerships

BreakAway Games610 420-8812

jmcnamara@breakawayltd.com