Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods

Ryan White All Grantee MeetingNovember 29,2012

Presenters: Mary Kay Diakite, LMSW and Graham Harriman, MAHIV Care, Treatment and Housing ProgramNYC Department of Health & Mental Hygiene

Disclosures• This continuing education activity is managed

and accredited by Professional Education Service Group. The information presented in this activity represents the opinion of the author(s) or faculty. Neither PESG, nor any accrediting organization endorses any commercial products displayed or mentioned in conjunction with this activity.

• Commercial Support was NOT received for this activity.

Disclosures•Mary Kay Diakite, LMSW

• Has no financial interest or relationships to disclose

•Graham Harriman, MA• Has no financial interest or relationships to

disclose

Learning ObjectivesAt the end of this activity, the participant will be able to:

1. Apply quantitative and qualitative methods in quality management program evaluations

2. Determine the most appropriate methods to apply in their particular EMA

Agenda•Background•Objectives•Methodology•Results

▫Quantitative▫Qualitative

•Discussion

Background• Quality management (QM) program goals

▫ Comprehensive performance measurement systems are developed & utilized to assess the quality of Part A services

▫ Enable Part A providers to continue to be skilled in using QI tools & methodologies to enhance the quality of their services

▫ Part A services improve health outcomes and quality of life for people living with HIV/AIDS

▫ The Part A QM Program remains responsive to the changing science & epidemiologic needs of the NY EMA community

Quality Management Activities

Organizational

Assessments

Performance Measuremen

t

Quality Management

Plans

Quality Learning Networks

Evaluation Objectives

Assess the Ryan White Part A QM Program conducted through the NYSDOH AIDS Institute during Jan-Dec 2011

Identify specific areas for improvement

Identify best practices & plan future trainings

Survey Overview

General perception

sAgenc

y

OALN

meetings

QI training

s

Performance

Indicators

Methodology & Timeline

Survey Development (Fall 2011)• Interagency reviewed & revised

Distribute Survey (Jan 30, 2012)• Distributed via SurveyMonkey to all QM staff at all provider agencies that participated in QM program in 2011

Analyze results (March-April 2012)• Quantitative—Descriptive statistics (%, ANOVA)• Qualitative – Open coding & Axial coding

Quantitative Results

Respondent Characteristics

Transitional Care Coordination (TCC)

Supportive counseling/family stabilization (SCF)

Outreach for homeless youth (OHY)

Mental Health (MSV)

Medical Case Management/Care Coordination- NYC (MCM)

Medical Case Management- Tri-County (MCM)

Harm Reduction Readiness & Relapse Prev (HRR)

Food and nutrition (FNS)

Early Intervention Services (EIS)

0% 5% 10% 15% 20% 25% 30%

1.2%

5.4%

1.8%

9.0%

28.1%

14.4%

20.4%

10.2%

9.6%

Percentage

Survey Respondents’ Characteristics

Community Health Center12.0%

Commu-nity

Based Orga-niza-tion

58.7%

Hos-pital

29.3%

Agency Type

• 7.93 years• Range = 0-25 years

Average Years

Worked

• 16.8% Executive Leaders• 67.1% Other• Program Director• Program Manager• Care Coordinator

Role at agency

Overall Satisfaction with QM Program*

1

1.5

2

2.5

3

3.5

4

4.5

5

3.80

4.13

3.623.95 3.76 3.86

3.33

3.66

4.50

Rat

ing

Aver

age

*No difference in means by service category, F = 0.98, p = 0.45

Overall 3.8

Satisfaction with Technical Assistance (TA)

1. No difference in means by service category, p > 0.052. TCC & OHY excluded because n<5

EIS

FNS

HRR

MCM NYC

MCM Tri-C

ounty MSV

SCF

1

1.5

2

2.5

3

3.5

4

4.5

5

3.73 3.67 3.64 3.683.89

4.10 4.00

TA helped create QM Plan 1,2

1

1.5

2

2.5

3

3.5

4

4.5

5

3.73 3.783.32

3.62 3.533.80

4.00

TA tailored to program needs 1,2

Overall3.58

Overall 3.75

Usefulness/Frequency of QM Activities

Very Useful

• LN Meetings • QI Trainings • Performance Review

Reports • Technical Assistance • Organizational

Assessments

Somewhat Useful

• Webinars/Teleconferences• Technical Assistance • Performance Review

Reports • Organizational

Assessments • LN Meetings

Not Useful

• Webinars/Teleconferences

• Organizational Assessments

• LN Meetings

Neither Increase/Decrease QM Activities• Communication (69.0%)• Webinars (66.7%)• QLN (64.3%)• Performance Monitoring (61.9%)• Coaching (54.8%)

Organizational Assessments (OA)

EIS HRR MCM NYC MCM Tri-County1

1.52

2.53

3.54

4.55

4.50

3.584.16 4.13

Feedback from OA helpful to program, N = 58

Overall3.94

1. No difference in means by service category, p > 0.052. TCC, FNS, MH, SCF removed because n<5

52.7%

12.7%

34.5%

Site receive OA, N = 110

YesNoNot sure

OA led to new strategies (N = 57)

HRR MCM NYC MCM Tri-County1

1.5

2

2.5

3

3.5

4

4.5

5

3.373.11

4.00

1. No difference in means by service category, p > 0.052. EIS, FNS, MH, SCF & TCC removed because n<5

Overall3.42

Usefulness of Performance Reports1,2

EIS MSV MCM-Tri-County FNS MCM-NYC HRR SCF1

1.5

2

2.5

3

3.5

4

4.5

5

3.71

3.33

3.91

3

3.84

3.12

4.25

N=60

Rat

ing

Aver

age

Overall3.58

1.No difference in means by service category, p > 0.052. TCC, OHY excluded n <5

Performance Reports, continued1,2

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

3.86

3.253.50

4.174.00 4.00

Relevancy of indicators

Overall3.85

EIS

FNS

HRR

MCM NYC

MCM Tri-C

ounty MSV

11.5

22.5

33.5

44.5

5

3.433.00

3.693.42 3.25

2.60

IPRO/NYCHSRO flexible

Overall3.42

1. No difference in means by service category, p > 0.052. SCF, TCC & OHY excluded because n<5

Performance Reports,

continued1,2

1

1.5

2

2.5

3

3.5

4

4.5

5

3.573.25

2.88

3.673.50

4.00

Reviews reflect performance

Overall3.38

1

1.5

2

2.5

3

3.5

4

4.5

5

3.14 3.00

2.44

3.583.50

3.00

Reports timely

Overall3.80

1. No difference in means by service category, p > 0.052. SCF, TCC & OHY excluded because n<5

Training topic Preferences• Quality improvement tools

1st Choice

• Communicating goals & Successes

• Planning & Implementing a QI project

• Using performance measurement

2nd

Choice

• Sustaining QI program3rd

Choice

4th Choice

• Gaining leadership support• Consumer involvement

Qualitative Results

Most Useful Webinars• n/a• None• Not useful

1st Choice30%

• Storyboard Development• Quality Learning Network

2nd

Choice8.5%

• Quality Management Planning

3rd Choice7.4%

4th Choice6.3%

• Quality Improvement Topics

Examples of New Approaches • PDSA• Storyboard

1st Choice11.5%

• Data Systems2nd

Choice7.6%

• Full agency CQI Team• Regular CQI Meetings

3rd Choice6.4%

4th Choice5.1%

• Retention Tool• QI as priority• New program forms

Quotes• We have made QM a priority and have a dedicated

steering committee that meets monthly.

• The program now has an eye to improving patient care and looking at specific goals to work on that we feel will improve patient care and health.

• I would say the main thing now is having the knowledge and skill to look at something, focus on it, and work towards improving it, and even if it does not generate the desired outcome, it is still a learning experience.

How LNL helped accomplish QI goals• Tools• Resources• Guidance• Framework

1st Choice24.6%

• TA on-site and on phone

2nd

Choice10.0%

• Not yet, just started• Support and feedback

3rd Choice8.7%

4th Choice7.2%

• Learn from other agencies

Quotes•The LNL provided guidance and

resources to develop and implement quality improvement strategies that were not available to members or that we did not have knowledge of. These tools were extremely helpful in improving our skills in quality improvement and shift our paradigm about determining quality measures and what to evaluate in our programs.

Quotes•The QLN inspired our QI team to utilize

fishbone diagrams when working on our QI project.

•The LNL came to our site and gave us feedback on how were doing overall.

Quotes•It helped us to think outside the box and

to separate what PHS expects from the program in terms of numbers with an overall and more general view on how services are provided and what is not working. We were then able to make appropriate changes to different areas of the program to make it more successful.

Additional Comments• n/a• None

1st Choice20%

• Helpful• Useful

2nd

Choice13.3%

• Good Networking3rd

Choice6.7%

Quotes•The learning network is a very helpful

networking program. I look forward to going there and know that I can get assistance with challenges that I may be experiencing in the program.

•It is a great opportunity to learn about what other providers are doing and also for us to share what has worked in our program.

How program used performance reports

• Helped us focus on one area

1st Choice20.5%

• Helped us improve our forms

• Led to QM strategies2nd Choice

11.4%

Quotes•Upon receiving the quality reports we

were able to focus on areas which needed improvement. One example was intakes/assessments. We were able to communicate our needs to other providers in which referrals were being funneled by several agencies and helped us to meet our goal.

Quotes•We were able to look specifically at our

screenings for STI and substance use to review for any possible edits and look at best practices when aligned with other agencies.

•Used reports to identify any program shortcomings and then developed a QM project to address the findings.

Ideas for Improvement - Themes

Meeting logistics

People should attend

regularly

Repetitive information

Setting annual goals for Learning

Network

Overlap of meetings

and groupsPerformance Reports

Meeting Logistics

Meeting Logistics

Time

Snacks

Location

Frequency

Meeting Logistics: Time•Meetings smack in the middle of the day

are very inconvenient

•It would be helpful if meetings could be scheduled to either start at the beginning of the day (9:00am or 9:30am) or end at the end of the day (4pm or 5pm)

•Meetings should be for a half-day only

Meeting Logistics, Cont.Frequency

• I believe that trainings should be offered twice a year rather than four or more times

Snacks

• Breakfast should also include (if any) cold drinks for those who do not drink or cannot drink coffee

Location

• Meetings should be done in Upper NYC or Bronx at times.

Regular Attendance

• The meetings might be more useful if agencies send the same people and each gets to present at least once yearly

Repetitive Information

• At times webinars and network meetings have felt somewhat redundant.

Setting annual goals for Learning Network

Also it would be helpful if there were annual goals for learning networks so that expectations for what will be accomplished for the year are clear to everyone participating (if sub-grantees are expected to have plans, so should the learning network)

Overlap of meetings and groupsWhen referring to the various quality improvement projects, it is extremely difficult figuring out what group of people you are referring to. For example, the HRR LN would be one group of people. Do they fall under the AI? Another group of people (NYSCHRO) usually comes in to collect data, are we referring to them? The UMRG sometimes facilitated by Susan Weigl, are you referring to the QI projects at this meeting? Too much overlap which makes it confusing to respond appropriately.

Performance Indicators and Reports• Select far fewer indicators and make sure they are

clearly defined. Right now, there are too many indicators, and it’s not at all clear that all programs define them in the same way. And whatever indicators are selected should be linked to program outcomes.

• NYCSHRO reviews require way too much time and attention and I don’t think the resources required to conduct them pay off in any way that’s beneficial to the portfolio.

• Performance reports given to us in a more timely fashion

Limitations•Low response rate (24.6%)

▫Response bias▫Generalizability

•People responding to questions having no experience with QLNs (OHY & TCC)▫We only included the Part A providers and

several Statewide providers are part of the LN--you should probably note that

Lessons Learned & Next Steps•Appropriate of QM activities•Review IPRO/NYCHSRO scheduling

procedures•Review reporting deadlines•Implement simple solutions

▫Meeting logistics

Questions? Thoughts?

Acknowledgements•Respondents•Interagency group

NYSDOH AIDS Institute: Tracy Hatton, MPH; Nova West, MPH; Susan Weigl, MPH;Johanna Buck, RN; Bruce Agins, MD, MPH

Public Health Solutions: Bettina CaroleRachel Miller, MPH

NYC DOHMH: Graham Harriman, MA ;Beau Mitts, MPH

Westchester Dept of Health: Tom Petro Julie Lehane, PhD

Obtaining CME/CE Credit•If you would like to receive continuing

education credit for this activity, please visit:

•http://www.pesgce.com/RyanWhite2012

THANK YOU!

Heather Mavronicolas, PhD, MPHDirector, Quality Management & Special ProjectsBureau of HIV/AIDS Prevention and Controlhmavroni@health.nyc.gov

Mary Kay Diakite, LMSWTechnical Assistance Project ManagerBureau of HIV/AIDS Prevention and Controlmdiakite@health.nyc.gov