performance assessment in simulation-based team training: principles from the science that matter
DESCRIPTION
Performance Assessment in Simulation-Based Team Training: Principles from the Science That Matter. Marissa L. Shuffler Eduardo Salas University of Central Florida February 11, 2010 Performance Testing Council Summit, Orlando FL. Overview. - PowerPoint PPT PresentationTRANSCRIPT
Performance Assessment in Simulation-Based Team Training: Principles from the Science That Matter
Marissa L. ShufflerEduardo Salas
University of Central FloridaFebruary 11, 2010
Performance Testing Council Summit, Orlando FL
Overview What do we know about teams and team
performance assessment? What is simulation-based team training (SBT)? How do we assess performance in team SBT? Best practices for SBT performance assessment
• Two or more individuals
• Multiple information sources
• Meaningful task interdependencies
• Coordination among members
• Common, valued goals
• Intensive communication
• Adaptive mechanisms
• Hierarchical structure
• Specialized member roles and responsibilities
• Task-relevant knowledge
Defining Characteristics of Teams
Driving Question:
How Do We Turn a Team of
Experts into an Expert Team?
The Conditions and Processes of Team Performance
Team Performance Assessment Systems
Requirements of a team performance measurement system (Cannon-Bowers & Salas, 1997): Must be theoretically based
Models of team performance Cognition, behavior and affective factors
Must consider multiple levels of measurement Individual/Team Process/Outcome
Must be able to describe, evaluate and diagnose performance Moment to moment changes Determine causes Help with remediation
Must support ease of use Easily adaptable Useability
Must have validity & good psychometrics Face External/Internal
Must facilitate Debriefing/AAR Feedback
Team Performance Assessment Systems
What We Know about Team Performance Assessment We have a good theoretical basis…
138 models! (Salas, et. al. in press) We have an idea (not perfect) of what to measure…
Team Type Taxonomies (Sundstrom et al, 2000) Team Competency Taxonomies (Cannon-Bowers et al,
1995) Shared Mental Models (Cannon-Bowers et al, 1993)
There are some tools available that work (not perfect)… Qualitative/Quantitative Outcome/Process Taskwork/Teamwork Used in Complex Simulations
What is Simulation? “Realistic synthetic work/learning environments
that can be used for training” (Andrews, 2005) What is Simulation-Based Training?
“A training strategy that uses several aspects of scientific, theory-based training, incorporating information, demonstration, and practice-based methods” (Salas, Priest, Wilson, & Burke, 2006)
Defining Simulation-Based Team Training
Defining Simulation-Based Team Training
What is Team Training? “A set of tools and methods that, in combination
with required competencies and training objectives, form an instructional strategy” (Salas & Cannon-Bowers, 2000)
Defining Simulation-Based Team Training
Used to accelerate expertise in team members…
Explicit links are developed and maintained among training requirements, exercise design, observation, and feedback…
Defining Simulation-Based Team Training
SBT provides trainees opportunities to demonstrate mastery of learned teamwork skills…
SBT provides opportunities to turn a team of experts into an expert team…
SBT Works Best With… …Clear & Precise Learning Outcomes …Carefully Crafted Scenarios …Well Designed Performance Measures …Robust Observation Protocols
Why Measure Performance in Team SBT? Quality measurement practices guide learning
and corrective feedback Measurement can also ensure learners possess
the requisite competencies for effective team performance
Practice opportunities in SBT can be used to detect deficiencies in both individual and team competencies
Needed Links in Simulation-Based Team Training
PerformanceHistory
Feedback
MeasuresScenarios &
EventsLearningOutcome
Tasks & KSAs
Challenges to Assessing Performance in Team SBT Measuring teamwork is complex!
Multiple people and multiple skills means multiple measures are required
Measuring the critical aspects of performance instead of measuring what is easy A single snapshot of performance may not be completely
representative of long term performance
It is difficult to remove observers or raters from the assessment process Raters are not always reliable!
How to Measure Team Performance in SBT Two important decisions from a training
perspective to guide assessment development: What meaningful feedback should be given to the
team and each individual member? What further training is required by team or
individual members? Depending on the answers to these questions,
certain methods may be more or less useful
Methods of Team SBT Assessment Event based measurement
Generates behavioral checklists that are linked to scenario events and KSAs being trained
Behaviorally Anchored Rating Scales (BARS) Provides brief descriptions of behaviors as
anchors associated with each particular rating
Methods of Team SBT Assessment Behavioral Observation Scales (BOS)
Generally uses a Likert type scale to rate the frequency of certain team processes
Self-Report Measures Questionnaires administered to each team
member individually
Best Practices for Team SBT Performance Assessment
Drawing from what we know regarding SBT and team performance assessment, several guidelines have emerged
These guidelines are theoretically, empirically, and practically based Derived from literature reviews, observations,
and interviews (Salas, Rosen, Weaver, Held, & Weissmuller, 2009)
Best Practices for Team SBT Performance Assessment
Best Practice 1: Ground measures in theory Use theory to answer questions about what to
measure Capture aspects of Input Process Output
models of team performance
Best Practice 2: Design measures to meet specific learning outcomes Clearly articulate the purpose of measurement at
the beginning of the development process Design measurement systems to capture
information necessary for making decisions about learning outcomes
Best Practices for Team SBT Performance Assessment
Best Practice 3: Capture competencies Explicitly link performance measures to
individual and team level competencies targeted for training
Best Practice 4: Measure multiple levels of performance Performance measures should be sensitive to
differences in individual and team performance
Best Practices for Team SBT Performance Assessment
Best Practice 5: Link measures to scenario events Link performance measures to opportunities to perform in
the simulation Affords greater structure & standardization for observers
Best Practice 6: Focus on observable behaviors Decreases “drift” of observers ratings over time Provides a fine level of granularity that can help in
providing feedback
Best Practices for Team SBT Performance Assessment
Best Practice 7: Incorporate multiple measures from different sources Different sources provide unique information Must develop a plan for integrating information
Best Practice 8: Capture performance processes in addition to outcomes Should provide information about HOW teams reached
their outcomes so that poor process can be corrected
Best Practices for Team SBT Performance Assessment
Best Practice 9: Create diagnostic power Provide information about the causes of effective and
ineffective performance
Best Practice 10: Train observers and structure observation protocols Establish training to ensure observers are accurately and
reliably rating performance Provide structured protocols to ease the information
burden of observation
Best Practices for Team SBT Performance Assessment
Concluding Remarks Team performance assessment is not easy, but
there is a wealth of research and theory that exists to support assessment tool development…so…
Use the science!
Thank You!
Questions?