assessment of ipe to move beyond attitudes: …...assessment beyond attitudes • intensifying...
TRANSCRIPT
Assessment of IPE to Move Beyond Attitudes: Featured tools and case studies
from the field Thursday, January 8
Moderator & Presenter
Sarah Shrader, PharmD
Clinical Associate Professor, University of Kansas, School of Pharmacy
Presenters
Erin Abu-Rish Blakeney, PhD, RN Faculty,
University of Washington, School of Nursing
Seattle, WA
Doug Brock, PhD Associate Professor
Department of Family Medicine and MEDEX Northwest,
University of Washington-School of Medicine
Seattle, WA
LuAnn Wilkerson, EdD Professor of Medicine and Senior Associate Dean for
Medical Education, David Geffen School of Medicine,
University of California, Los Angeles
Assessment of IPE to Move Beyond Attitudes: Featured Tools and Case
Studies from the Field
Part 1
January 8, 2015
Sarah Shrader, PharmD, FCCP, BCPS, CDE (moderator)
Assessment Beyond Attitudes • Intensifying interest to assess IPE beyond attitudinal
measures
• Quantitative, qualitative, and mix-methods approach
• Cochrane Review Updated 2013: “To improve the quality of evidence relating to IPE and patient outcomes or healthcare process outcomes, the following three gaps will need to be filled: first, studies that assess the effectiveness of IPE interventions compared to separate, profession-specific interventions; second, RCT, CBA or ITS studies with qualitative strands examining processes relating to the IPE and practice changes; third, cost-benefit analyses.”
Reeves S, Perrier L, Goldman J, Freeth D, Zwarenstein M. Interprofessional education: effects on professional practice and healthcare outcomes (update). Cochrane Database of Systematic Reviews 2013, Issue 3. Art. No.: CD002213. DOI:10.1002/14651858.CD002213.pub3.
Evaluation Outcomes Measured
Abu-Rish, E. et al. Current Trends in Interprofessional Education of Health Sciences Students: A Literature Review. 2012;26: 444-51.
Assessment Methods
Abu-Rish, E. et al. Current Trends in Interprofessional Education of Health Sciences Students: A Literature Review. 2012;26: 444-51.
A Point to Ponder
• Interprofessional Activity/Program Evaluation VS. • Interprofessional Learner Assessment
Level 1a: Reaction • Learners’ views on the learning experience and its interprofessional
nature Level 2a: Modification of attitudes/perception • Changes in reciprocal attitudes between participant groups.
Changes in perception or attitude towards the value and/or use of team approaches to caring for a specific patient/client group.
Level 2b: Acquisition of knowledge and/or skills • Including knowledge and skills linked to interprofessional
collaboration. Level 3: Behavioral change • Identifies individuals’ transfer of interprofessional learning to their
practice setting and their changed professional practice Level 4a: Change in organizational practice • Wider changes in the organization and delivery of care Level 4b: Benefits to patients/clients • Improvements in health or well-being of patients/clients
Kirkpatrick/Barr’s Evaluation Framework
From Barr, Koppel, Reeves, Hammick, Freeth. Effective Interprofessional Education Argument, Assumption & Evidence. 2005 Malden, MA: Blackwell Publishing,Ltd.
Miller’s Pyramid of Assessment
Additional Resources Canadian Interprofessional Health Collaborative. An inventory of quantitative tools measuring interprofessional education and collaborative practice outcomes. 2012 Aug. Available from http://www.chd.ubc.ca/news/featured-inventory-quantitative-tools-measure-interprofessional-education-and-collaborative-pra. National Center for Interprofessional Practice and Education Measurement Instruments https://nexusipe.org/measurement-instruments
Additional Resources Valentine MA, Nembhard IM, Edmondson AC. Measuring teamwork in health care settings: A Review of Survey Instruments [Internet]. Harvard Business School. 2011 May. Report No.:11-116. Available from: http://hbswk.hbs.edu/item/6727.html. Measuring the Impact of Interprofessional Education on Collaborative Practice and Patient Outcomes: A Consensus Study. October 7, 2014. Available from: https://www.iom.edu/Activities/Global/MeasuringtheImpactofInterprofessionalEducation/2014-OCT-07.aspx
Additional Resources
Blue A, Chesluk B, Conforti L, Holmboe E. Assessment and Evaluation in Interprofessional Education: Lessons Learned from a Multi-Methods Study. • Article accepted for publication in the Journal
of Allied Health • Webinar from 2014 posted on National Center
Resource Exchange (click link above).
ASSESSMENT OF IPE TO MOVE BEYOND ATTITUDES: VALIDATION OF THE PERFORMANCE ASSESSMENT FOR INTERPROFESSIONAL COMMUNICATION AND TEAMWORK (PACT)
Doug Brock, PhD Erin Abu-Rish Blakeney, PhD, RN Chia-Ju Chiu, PhD, PT
Acknowledgements • This work was completed as Dr. Chia-Ju Chiu’s dissertation, to develop widely applicable tools for assessing team performance.
• The work was supported by the University of Washington Macy Project Evaluation Workgroup.
• The work was funded in part by a grant from the Josiah Macy Foundation.
Background
• University of Washington IPE
• Training pre-licensure students
• Focus on Kirkpatrick/Barr acquisition of skills / knowledge
• Assessment of team performance, not individuals
• Emphasis on simulation
• Josiah Macy Foundation Grant
• TeamSTEPPS® National Training program (2009)
Conceptual Framework– TeamSTEPPS®
Aims 1) Develop psychometrically sound measures to
assess teamwork and communication skills
2) Collect data, summarize and present construct validity evidence for the PACT tools
Context—The Macy Project • Three simulated acute care cases aimed at improving team communication skills • Dyspnea in a Hospitalized Patient • Teenager with Asthma • Postoperative patient with tachycardia
• Medical school capstone included medical, nursing, pharmacy, & PA students • Pilot study—49 IP students • Validation study—306 IP students
Development of the PACT
PACT Toolset Validation Methods • Tools included in validation study
• PACT-Novice – Peer or untrained observer scale • PACT-Expert – Faculty or expert scale • PACT-Video – Trained raters
• Assessed reliability and validity
PACT—Expert
PACT—Novice
PACT—Video
Construct Validity Evidence 1) Content 2) Substantive 3) Structural 4) Generalizability 5) External 6) Consequential * Messick, 1995
Reliability Evidence
Version Number of
Valid Records
Number of Items
ICC F Cronbach’s
Alpha
PACT-Novice
904 5 .85 6.46** .85
PACT-Expert
85 13 .76 4.09** .79
PACT-Video
112 26 .90 9.63** .91
ICC(1,k) was calculated with one-way random effects model where rater effects are random. ICC for Average Measures was reported. ** F is significant p < .001
Content: Relevance & Representativeness • Scenarios, training, & tools were developed incorporating TeamSTEPPS® framework, constructs & terminology:
1. Team Structure,
2. Leadership,
3. Situation Monitoring,
4. Mutual Support,
5. Communication.
• All 5 TeamSTEPPS teamwork constructs were represented in global quality ratings in all tools • PACT-Expert & PACT-Video included behavioral markers
Substantive: PACT-Novice & PACT-Expert Consistency in Responses & Processes • PACT-Novice scoring system was introduced to the raters during didactic session.
• Scenarios unfolded under faculty guidance.
• Coaching of clinical skills ensured focusing on communication and teamwork skills.
• Administered immediately following simulated scenarios & before the debriefing.
Substantive: Consistency in Responses & Processes PACT-Video • PACT-Expert raters were trained through PACT-Video rater training program & were experts in TeamSTEPPS®.
• PACT-Video raters received rater training program before rating the videos.
• Strict rating protocol was developed to ensure the consistency of rating processes.
Structural: Scales reflect theoretical target
• Internal consistencies of the subscales • Exploratory factor analysis
Structural—PACT Expert Behavioral Marker
Rotated Factor*
Communication Leadership & Team Structure
Mutual Support
Hand-off &
S-Bar 10. Repeat back instructions or clarifications. .673
5. Applies the STEP process. .627 .355
13. Demonstrates closed-loop communication. .597
12. Demonstrates SBAR. .544 .520
9. Verbalize their activities aloud. .524
2. Refer to protocols and checklists. .362
3. Conducts briefs/huddles/debriefs. .326 .565
1. Recognize a leader .769 4. Empowers team members to speak freely and ask questions.
.658
6. Patient/Family is included. .363
8. Use Two Challenge rule or CUS. .889
7. Ask for assistance prior to task overload. .353
11. Hands off to another team member. .597 Extraction Method: Principal Axis Factoring. Rotation Method: Varimax with Kaiser Normalization.
*Only values > 0.3 are displayed.
Conclusions • Validity is a continuing and iterative process of establishing evidence to support specific uses & interpretations of the tool.
• Preliminary construct validity evidence for PACT was demonstrated.
Next Steps • Broad dissemination of PACT toolset
• American Educational Research Association (AERA) 2015 • Additional publications
• Strengthen validity: • Replication/Different study designs to enhance generalizability • Build external validity through associations with external
measures • Explore the consequences/impact—consequential validity—of
team assessment
• Explore use with practicing clinical teams
Best Uses for PACT Toolset • TeamSTEPPS-aligned situations/scenarios
• Assess team performance (not individual)
• PACT-Novice: During teaching/training • Observers • Debriefing
• PACT-Video: Research tool for understanding team behavior and performance.
PACT Toolset Resources • Link to PACT tool set: http://collaborate.uw.edu/educators-toolkit/tools-for-evaluation/performance-assessment-of-communication-and-teamwork-pact-too
• Link to “Simulation Team Training Toolkit”: http://www.collaborate.uw.edu/educators-toolkit/interprofessional-simulation-team-training-faculty-toolkit/simulation-team-trainin
Creating Innovative Tools for Assessing Learners’ Interprofessional
Competencies LuAnn Wilkerson, EdD
With funding from the Josiah Macy Jr. Foundation And a team of co-investigators from Nursing and Medicine
Assessing Interprofessional Competencies
IPEC Core Competencies
Assessment Tools Values/ Ethics
Roles/ Responsibilities
Interprofessional Communication
Teams/ Teamwork
OSCE stations X X X
Direct Observation X X
Knowledge Test X X X
Video Case Exam X X X
Implicit Association Test X X
Multi-Source Evaluation X X
Assessing Interprofessional Competencies
IPEC Core Competencies
Assessment Tools Values/ Ethics
Roles/ Responsibilities
Interprofessional Communication
Teams/ Teamwork
OSCE stations X X X
Direct Observation X X
Knowledge Test X X X
Video Case Exam X X X
Implicit Association Test X X
Multi-Source Evaluation X X
OSCE Station
Roles/Responsibilities Interprofessional communication
Teams/Teamwork
40
• Setting: Outpatient heart failure follow up clinic • Complaint: need for medication change • Roles: NP and senior medical student
– Standardized NP and senior medical student OR – Standardized senior medical student and NP student
• Can be used as a simulation or an OSCE
OSCE Station
Roles/Responsibilities Interprofessional communication
Teams/Teamwork
• Setting: Outpatient Family Practice Clinic • Complaint: sudden change in vision • Roles: NP student and medical student • Checklist focus:
– Skills in handoffs – Collaborative decision making – Understanding scope of practice
Interprofessional Collaborator Assessment Rubric – Individual performance
TeamSTEPPS Questionnaire – team performance
eWalk© Direct Observation Tool Interprofessional Communication
Teams/Teamwork
eWalk® (Media-X Systems Inc., Ottawa, Canada) uses a tablet database to collect, store, analyze, and report observational data
eWalk® (Media-X Systems, Inc, Ottawa, Canada) allows to collect, store, analyze, and report data based on real time observation.
Knowledge Test
Roles/responsibilities Interprofessional communication
Teams/teamwork
• Item bank for test construction – 49 questions keyworded to 37 discreet objectives from the
Core Competencies for Interprofessional Collaborative Practice
• Vignette-based • Multiple choice format • 7-item collaborative practice case study
– 20 questions on system-based practice and policy
• Answer key
• Goal: to synthesize feedback from team members with whom a student has worked for a period of time
– Student submits the names of 6 team members – Online survey tool (REDcap) is used to contact and remind invitees to respond
• Survey uses 6 Interprofessional Collaborator Assessment Rubric categories – Communication (7 items) – Collaboration (4 items) – Roles and Responsibilities (7 items) – Collaborative Patient-Family Centered Approach (4 items) – Team Functioning (5 items) – Conflict Management/Resolution (4 items)
• Invitee selects which behaviors he/she has observed and indicates level of the described behavior demonstrated over time
– Developing -- Occasionally demonstrates – Competent -- Frequently demonstrates – Mastery -- Consistently demonstrates
Multi-source Assessment
Interprofessional Communication Teamwork
Interprofessional issues addressed in The Patient or the Protocol: • Shared decision
making • Teamwork • Conflict of interest • Paternalism • Palliative care • Roles and
responsibilities • Clinical trials and
informed consent • Patient safety
Zaption® (Stigler JW, Zaption Inc.,2013 ) combines video with question/response database https://www.zaption.com
Video Case Exam
Values/Ethics Roles/Responsibilities
Teams/Teamwork
Zaption® can add to any online video: • multiple lines of text • Images • drawings done with
the mouse • essay questions with
multiple-line text box • questions requiring a
number for response • questions requiring
drawing the response • multiple choice
questions
47
• The IAT was developed by Tony Greenwald and Mahzarin Banaji in 1994 to study attitudes and beliefs that are “normally outside of conscious control.” – Measures the strength of associations between concepts and
evaluations or stereotypes about others. – In the early 2000’s, Nosek, Greenwalk and Banaji developed a
demonstration website to study intergroup attitudes and stereotypes. – Growing evidence that these unintended biases can be changed. – https://implicit.harvard.edu
Banaji & Greenwald, The Blind Spot: Hidden Biases of Good People, 2013.
Implicit Association Test
Roles/responsibilities Values/ethics
• Goal: help students evaluate their implicit assumptions about the roles of nurses and physicians in the health care system.
• The concepts and evaluative terms for the IAT were chosen based on focus groups held with UCLA medical students and advanced practice nursing students.
• Project Implicit created the interprofessional IAT.
Formative Evaluation Evaluation Questions Methods
1. How do the results of the student needs assessment inform tool design, content and learning outcomes?
Focus groups with Advanced Practice nursing students and 3rd year medical students
2. What are the major competency gaps for medical and APN students and do they differ by students’ clinical level or profession?
Same as above
3. How useful are the assessment tools when used formatively?
Pilot testing 6 tools
Implementation Evaluation Evaluation Questions Methods 1. Usability: How easily do students find the IPE assessment tools to complete? How can the tools and implementation be improved?
Develop usability evaluation for each tool
2. What improvements can be made in the faculty and preceptor implementation guides for the assessment tools to better prepare for developing and/or observing IP competencies?
Pilot test each tool
3. How can the implementation processes be improved or refined to optimize the product?
Pilot test each tool
4. What are the psychometric properties of each of the assessment tools? Are revisions needed to improve reliability and validity?
Analyze pilot data
Outcome Study Methods Evaluation Questions Cohort study
1. To what extent do students’ performance on the new assessment tools vary based on prior experience in IP settings, prior academic performance, age, gender, health profession, or mean ratings of their SBH small group instructors?
2. To what extent does classroom team composition of System-Based Healthcare course nursing and medical students moderate results on the assessment measures? 3. Do medical students trained in IP groups differ from those trained in medical-student-only groups on the OSCE stations? 4. What are the strengths and major threats to validity related to the study design?
Assessing Interprofessional Competencies
IPEC Core Competencies
Assessment Tools Values/ Ethics
Roles/ Responsibilities
Interprofessional Communication
Teams/ Teamwork
OSCE stations X X X
Direct Observation X X
Knowledge Test X X X
Video Case Exam X X X
Implicit Association Test X X
Multi-Source Evaluation X X
54
Questions?
Save the Date
Assessment of IPE to Move Beyond Attitudes: Featured tools and case studies from
the field
Presented by Amy Blue, PhD,
Vernon Curran, Phd and
Sara Shrader, PharmD, FCCP, BCPS, CDE
Thursday, February 12
10:00 – 11:00 A.M. Central Time
Thank you!
Please visit us at www.aihc-us.org for more information on future AIHC Webinars.