games to improve clinical practice and healthcare administration
TRANSCRIPT
Gaming for Organizational, Clinical and Patient Risk Management
Comprehensive Unit-based Safety Program (CUSP) for Enhanced Recovery Protocol for Surgery
“Simu-Leader” (a gaming application) YouTube link
What were we trying to do?Johns Hopkins Medicine implemented the Comprehensive Unit-based Safety Program (CUSP) and has demonstrated significant outcomes. Armstrong Institute conducted a brief table top game in 2012 to facilitate learning and reinforce best practices amongst hospital leadership. Based on those results a grant was requested to build an automated application to achieve the same results on a larger scale.
Two developed are:
1. Patient Risk: Application geared to improve the safety and quality of acute patient care , strongly impacted by all levels of organizational leadership. Game provides evidence-based methods for developing the knowledge and skills necessary for effective safety leadership
2. ERAS/CUSP: Gamification and gaming interfaces measurably improve patient outcomes by implementing enhanced recovery after surgery (ERAS) practices in hospitals through applying the Comprehensive Unit-based Safety Program (CUSP).
Overall Study Aims§ Specific Aim #1: Evaluate the comparative effectiveness of simulation and traditional methods of management instruction on multi-level learning outcomes.
Ø H1a-c: Compared to traditional instruction, simulation will produce significantly higher individual level a) learner reactions, b) learning, and c) behavior change.
§ Specific Aim #2: Compare the impact of ad hoc and intact management team configurations on multi-level learning outcomes and the quality of strategy generation in organizational leadership simulations.
Ø H2a-c: There will be a significant difference due to the team configuration manipulation for individual level a) learner reactions, b) learning, and c) behavior change.
Ø H3a-c: There will be a significant difference due to the team configuration manipulation for team level creativity of ideas generated in terms of a) novelty, b) usefulness, and c) feasibility of ideas.
§ Specific Aim #3: Evaluate the impact of a mindfulness pre-training intervention on multi-level learning outcomes and management team performance. Pre-training and pre-performance interventions are effective, efficient ways to enhance learning and performance outcomes.
Ø H4a-c: Compared to no mindfulness training, mindfulness training will increase the efficacy of the organizational leadership simulation by producing significantly higher individual level a) learner reactions, b) learning, and c) behavior change.
Ø H5a-c: Compared to no mindfulness training, mindfulness training will produce significantly higher team level creativity of ideas generated in terms of a) novelty, b) usefulness, and c) feasibility of ideas.
ERAS/CUSP Overall Aims
Task 1: Develop a public domain version of a CUSP for ERAS educational kit for national distribution.
Ø Johns Hopkins APL to leverage software engineering expertise as well as existing applications to address the requirements. APL will design, develop, test and deliver the software application
Task 5: Assess the Adoption of CUSP for ERAS and Evaluate the Effectiveness of the Intervention in the Participating Hospitals.
Ø Johns Hopkins APL will develop software and provide a credible and systematic data collection schema allowing for data collection. The software application will provide a robust back-end database structure for in-stride assessments, metadata visualization and distributed access.
Ø Subtask 5.1: Design and develop data collection protocols and analysis plan. Support the data collection protocol provided by AI collaborators. Make software application available to participating hospitals to voluntarily input and share data.
Key Metrics, Measures & Performance
How$Represented?$
How$Collected?$
Expected$Format$
Weigh:ng$
Transparency*
Reliability*
Leadership*
Strategic*Plans*
Impact*
8*8*8*other*metrics*
Performance in the application was measured in a variety of ways:• Individual progress + translation to practice• Team effectiveness within scenario(s)• Between team comparisons conditioned on mindfulness training
Armstrong Institute will define each metric for JHU/APL to include
Teams Team v TeamMeasures
Cond v CondMeasures
Candidate metrics for response/resolution:• Rate of re-admission• Patient falls• Acquired infections• Preventable VTE
Detailed design connecting data needs to GUI
Software Design & Computer Science
Technical Collaboration
As elements of the application are developed in parallel by Johns Hopkins Medicine, Armstrong Institute and JHU/APL, we anticipated evolving requirements. Therefore, the Agile software development methodology best suited the development team.
“Agile software development is a group of software development methods in which requirements and solutions evolve through collaboration between self-organizing, cross-functional teams. It promotes adaptive planning, evolutionary development, early delivery, continuous improvement, and encourages rapid and flexible response to change.”
The web application is updated frequently during the development process in order to facilitate frequent feedback from collaboration discussions.
Participant Rhythm1
Player Feedback
Software Architecture
Client Layer
Amazon Cloud Server
Windows Server 2012
MS IIS Webserver
Production Web App
MS SQL Server Database
Web App
Game ManagerGame Session
Game Session…
Game Session
Security Services
Game Engine
Text Chat
Player Interface
Principal Inv. I/F
Performance Measurement Reporting
Web Page Access
Spreadsheets
PDFs Database
Designed Reports
Game Navigation ConceptConcept discussions with Armstrong Institute produced a hub-spoke structure which migrated to a more functional physical representation of users’ expected normal work environment.
Shown here is early concept art used to scope the application and conduct early usability tests.
User Interface via Web Application (prototype)
Web Portal on Amazon Web Services
Software Demonstration
Human Factors & Usability
Human Factors Tasking
In Progress: CompletedRound 1, N=9 Round 2, N=9
Completed N=4
Human Factors TaskingConcept Vetting - Preparation
Human Factors TaskingAlpha Testing & Usability Testing
Background Questionnaire
Proctored Activity Scenarios
Usability Questionnaire
Retrospective Interview
Human Factors TaskingAlpha Testing & Usability Testing
Background Questionnaire
Proctored Activity Scenarios
Usability Questionnaire
Retrospective Interview
Human Factors TaskingAlpha Testing & Usability Testing
Background Questionnaire
Proctored Activity Scenarios
Usability Questionnaire
Retrospective Interview
Human Factors TaskingAlpha Testing & Usability Testing
Background Questionnaire
Proctored Activity Scenarios
Usability Questionnaire
Retrospective Interview
What did the participants find important during the session?What problems did they encounter? (Flanagan, 1954).
Their perception of the session and their self-reflection may offer insight into how the simulation was used and how it could be improved (Rosson & Carroll, 2002).
Questions included:• Are the instructions clear?• Do you feel you know what to do/where to navigate?• What would make this easier for you?• Can you recall any activities being particularly successful? Unsuccessful?• What did you like the most? Dislike the most?
Human Factors TaskingAlpha Testing & Usability Testing – Results (3/3)Post-Session Usability Questionnaire
• Round 1 – July (purple), n=9. Round 2 – November (blue), n=9. • Four participants completed both rounds.
Metrics and Verification
Scoring Algorithm
§ Outcome = Pos. tactics + Neg. tacticsØ Pos. Behaviors = (6RCpos + 3TRpos + BCpos) * ((TL +1) + BS)Ø Neg. Behaviors = (-2.2RCneg + -3TRneg + -1.6BCneg) * (TL +1)
§ Term definitions/MOPS/Tactic DomainsØ RC = Responsibility, role clarity, and feedback tacticsØ TR = Time and resources tacticsØ BC = Building capacity tacticsØ TL = Transformational leadership tacticsØ BS = Boundary Spanning tactics
§ Term values will be based on proportion of (+) and (-) actions selected during game (TL & BS will be a count of actions)
§ Initial weights determined through adapted ProMES process
How much does each indicator contribute to effective organizational performance?
Three primary types of contingency shapes:
-100-80-60-40-20020406080100
80 85 90 95 100PERCENT PASSING INSP.
EFFECTIVENESS
Diminishing Returns
-100-80-60-40-20020406080100
80 90 100 110 120 130TRAINING MET
EFFECTIVENESS
Linear Critical Mass
Images: (Pritchard, 2009)
Selecting Different Tactic Sets
5 7 9 11 13 15 17 19 21 23 25 27
050
150
250
# of Tactics Selected
SSI
5 7 9 11 13 15 17 19 21 23 25 27
0100
200
300
# of Tactics Selected
RA
5 7 9 11 13 15 17 19 21 23 25 27
0100
200
300
# of Tactics Selected
LOS
5 7 9 11 13 15 17 19 21 23 25 27
050
150
250
350
# of Tactics Selected
Sat
Surgical Site Infections Patient Readmissions
Length of Stay Patient Satisfaction
Impact of “positive” and “negative” tactics
positive negative overall
-50
050
100
150
200
250
300
Surgical Infections
positive negative overall
0100
200
300
Readmission
positive negative overall
0100
200
300
Length of Stay
positive negative overall
050
100
150
200
250
300
350
Patient Satisfaction
Risks for our small budget game§ On-demand help is not available during game play increasing the level of software reliability and testing
§ Technology barriers that exist at the player location, equipment or access
§ Depth and/or breadth of scenario data required to represent a realistic decision environment
§ Voice communication implementation for system users
§ Adjudication formulation represents ‘real’ hospital outcomes
§ No timing mechanism included to compel player action
Summary
Major Take-AwaysØ Everyone wants gaming and it is ‘accepted’Ø Professional game success is measured differentlyØ Represent “real life” with plausible actions and effectsØ Technical skills and experimentation over graphicsØ Experimental testing focuses on impact over entertainment
The game and initial results perform well against the study hypotheses. Participants from around the world have registered and data collection is on-going. We anticipate
significant engagement and persistent effects for individuals and organizations participating.
Scott Simpkins;; [email protected];; 240-228-3718Principal Staff, Principal InvestigatorSupervisor, Health Concepts & Systems Analysis
Health Systems Engineering Group
Patient Risk Management Summary
The safety and quality of acute patient care is strongly impacted by all levels of organizational leadership, yet few evidence-based methods for developing the knowledge and skills necessary for effective safety leadership currently exist. Simulation clearly improves technical and non-technical competencies of healthcare workers, as well as organizational learning and continuous improvement. However, simulation has not yet been broadly applied to the development of leadership for patient safety.
This work assembled a multi-disciplinary team with unique and diverse expertise (simulation, training, gaming, engineering, patient safety leadership, business and management, social science of creativity, human factors and organizational psychology), and capitalizes on existing projects and strong relationships with state, national and international healthcare organizations to evaluate a game for patient safety leader development.
ERAS/CUSP SummaryTo measurably improve patient outcomes by increasing the implementation of enhanced recovery after surgery (ERAS) practices in hospitals, through the use of an adaptation of the Comprehensive Unit-based Safety Program (CUSP). CUSP for ERAS shall be developed as an integrated combination of clinical and cultural interventions. § Develop a consensus description of the essentials of ERAS, develop consensus recommendations for implementation of ERAS bundles for various surgeries and determine accurate measures of effectiveness of CUSP for ERAS.
§ Develop an adaptation of the CUSP protocol and materials to be applied to ERAS.
§ Recruit hospitals to implement ERAS in a phased approach, beginning with 100 hospitals in the base period, and expanding to include approximately 200-250 hospitals during each option period, in order to cumulatively include 750 or more hospitals. Participating hospitals should be from all States, Puerto Rico and the District of Columbia.