current examples of joint hsr/sr evaluation edward j. miech, ed.d. research scientist health...

Current Examples of Joint HSR/SR Evaluation

Edward J. Miech, Ed.D.Research Scientist

Health Services Research & DevelopmentVA-CASE (VISN11 VERC)

Roudebush VA Medical CenterIndianapolis, IN

July 14, 2010

VA Cancer Care Collaborative, Phase II

• 21 VA Teams Working on 5 Different Cancer Types– Lung, Prostate, HCC, Head & Neck, Colon

• SR-based Collaborative• Each team has a coach who is experienced with

SR, team dynamics, VA context

VA Cancer Care Collaborative, Phase II

• Each site receives monthly visits by an Industrial Engineer

• Learning Sessions include sessions on SR principles, concepts & tools

• Successor to VA Cancer Care Collaborative-Phase I

• Co-Chairs: Ken Klotz and Kim Voss

Team Development Measure

• CCC II planning committee interested in learning more about team dynamics

• In May 2010 (during Action Period 1) individual CCC II team members were invited to complete online version of Team Development Measure

• n=97; 18 Teams ≥ 3 members completed TDM

Team Development Measure

• 31-item instrument where respondents rate level of agreement with statements about team cohesion, communication, roles and goals

• Respondents retain anonymity• Takes 5-10 Minutes To Complete

Team Development Measure

• Examples of Statements– Team members say what they really mean. – The goals of the team are clearly understood by

all team members. – The work I do on this team is valued by the other

team members.

• 4-point Likert response scale: – Disagree Strongly, Disagree, Agree, Agree Strongly

Team Development Measure

• Permission Granted To Use Freely In VA• VA Guide To Using TDM Developed Under

Leadership of DeDe Ordin and Brian Mittman – 12-page detailed handbook, includes questions– published February 2010

• Freely Available on QUERI website at http://www.queri.research.va.gov/ciprs/TDM_Module.pdf

Team Development Measure

Preliminary Results

Team Development Measure

• Coaches Receive Customized Reports For Each of 18 CCC II Teams– Team Reports will include distribution of

individual responses for each TDM item– Visual depiction of members’ different

perceptions of current state of team

Team Development Measure

Example of TDM Customized Team Report

Number of respondents = 6Mean score = 65Lowest score = 50Highest score = 88 The graph below shows how many team members see the team at what stage.

Baseline, June 2010

INSERT SNAG IT OF TDM HERE

INSERT SNAG IT OF TDM HERE

INSERT SNAG IT OF TDM HERE

Team Development Measure

• Coaches have option to lead 1-hour facilitated discussion

• Facilitated discussion led by coach– Where team is now– What next stage looks like– How the team might get to next stage

Team Development Measure

• Data captured for TDM is quickly analyzed and reported back to CCC planning committee

• TDM Data Cross-Referenced With Other Measures

• At level of team, 1. diverse perceptions of how members see the

team leads to 2. an evidence-based discussion that3. promotes further team development and growth

Team Development Measure

• Tool = Online Instrument (Kwik Surveys)• All-Digital Data Capture and Organization– Downloads automatically into Excel

• Respondents Retain Individual Anonymity – Allows for candid responses to questions

• Multi-Level Data– Tracked at both individual and team level– Summary Reports Useful at both Collaborative

and Team levels

PCMH Collaborative LS1 - West

• Learning Session with approximately 300 participants taking place right now in San Diego

• 300-clicker ARS system in use to capture baseline and endpoint data (i.e., pre- and post-)

Baseline ARS at PCMH LS1 - West

Selected Items

How many collaboratives have you participated in, including this one?

1 2 3 4 5 6

41%

36%

10%

4%3%6%

1. 12. 23. 34. 45. 56. 6 or more

How many years have you worked for the VA?

1 2 3 4 5 6 7

7%

16%18%

8%

16%

14%

21%

1. < 12. 1 - 23. 3 - 54. 6 - 105. 11 - 156. 16 - 257. 26 +

What is your job title?

1 2 3 4 5 6 7 8 9 10

18%

1%

8%

26%

9%

2%3%

18%

6%

8%

1. Administrative Assoc2. HPDP/BHC3. LVN/LPN4. Nurse5. PCMH Faculty & Staff6. Pharmacist7. Physician8. Social Worker9. Technician/MA10. VA Faculty Leader

How many VA project-based teams (QI, Systems Redesign, Lean, etc.) have you personally been a

member of, including this one?

1 2 3 4 5 6

26%

20%

29%

3%

8%

15%

1. 12. 23. 34. 45. 56. 6 +

Did you complete the prework before you came to LS #1?

1 2 3 4 5 6

54%

15%

4%2%

10%

16%

1. Yes, all2. Yes, most3. Yes, some4. No5. Did not know about

pre-work6. N/A

I had enough knowledge about the tools available to complete the prework.

1 2 3 4 5

12%

34%

5%

18%

31%1. Strongly Agree2. Agree3. Neutral4. Disagree5. Strongly Disagree

I had enough resources, including time, to complete the prework.

1 2 3 4 5

7%

20%23%

41%

10%

1. Strongly Agree2. Agree3. Neutral4. Disagree5. Strongly Disagree

Exploratory Data Analysis: Correlations

• “How many Collaboratives have you participated in?” and– How many project-based teams have you been

on? (r = .55, p<.0001)– Did you complete the LS1 pre-work? (r = .21

p<.001)

ARS data from CCC II LS1-New Orleans

• Statistically significant correlations between participation in any prior VA Collaborative and– Higher levels of self-reported familiarity with

Systems Redesign tools (r=.59, p>.0001)– Higher levels of perceived support from site

leadership for Collaborative work (r=.42, p<.02)– Higher letter grades given by participants to LS1

(r=.41, p<.02)

Common Features: ARS & Online TDM

• All-Digital Capture (with built-in reporting features)

• Respondents retain anonymity• Responses can be tracked at individual level• Data can be segmented, aggregated, analyzed

for correlations, etc.

VA Yellow Belt Training Program

• In VA Yellow Belt training program participants actively learn about Systems Redesign and Lean principles, concepts, and tools through simulations, group application exercises, didactic presentations, and discussion

• Enrollment capped at 50 participants per session • Courses usually taught by faculty from the

Purdue PharmaTAP program • Sessions generally take place for 3.5 consecutive

days in an off-site location such as a hotel

VA Yellow Belt Training Program

• Over 400 participants in VA Systems Redesign Yellow Belt training across 12 different sessions held between August 2009 (pilot in Indianapolis) and June 2010 (Boston, MA)

• Pre- and post-data collected at 11 of these 12 sessions

Example: Interim YB Findings

Example: Interim YB Findings

• Analyses found extremely strong correlations between:– the items “I believe this Yellow Belt workshop will

be relevant to my job” and “I am positive about Systems Redesign” (r=.68, p<.0001);

– the items “I found this workshop to be practical” and the overall letter grade given to the workshop (r=.67, p<.0001)

Example: Yellow Belt in Connecticut

• I am familiar with many of the tools used in Systems Redesign. (n=34)

ARS Advanced Features

• Multi-Voting• Priority Ranking – 1st choice =10 points– 2nd choice =5 points– 3rd choice= 1 point

• Impact-Effort Matrix– Graph two dimensions along x-axis and y-axis

• Conditional Branching

Next Frontier

• Capturing Multi-Level Data From Complex, Multi-Site Initiatives in a Single Project File

• Flexibly Interrogating/Analyzing Evidence– Keywords– Codes– Indexing

Next Frontier

• Adding Value to Source Data with – Applying codes and tags– Linking to Analytic Memos– Performing Context Matrix Analyses

• Rapid Reporting of findings/results to initiative organizers/planners on rolling basis

• Capturing, Organizing, Analyzing and Reporting Rich-Context Data occurs on ongoing, rolling basis

• Digital Tool = NVivo8

Acknowledgements

Heather Woodward-Hagg– Systems Redesign, Chief, Roudebush VA– Co-Director, VA-CASE (VISN11 VERC)

Mary Sherrill– Co-Director, VA-CASE (VISN11 VERC)

Dawn Bravata– Co-Coordinator, Stroke QUERI