australasian evaluation society conference sydney august-september 2011 strength of partnerships: a...
DESCRIPTION
Role of Government Governments have an important role in creating or enabling the foundations of effective systems for organised primary health care service delivery (Takach, Gauthier, Sims-Kastelein, & Kaye, 2010; World Health Organisation, 2008). This requires deliberate and sophisticated policy processes that maximise the potential within complex systems and the capacity within these systems for innovation (McDaniel & Driebe, 2001; Swantson, et al., 2010; World Health Organisation, 2008).TRANSCRIPT
Australasian Evaluation Society ConferenceSydney August-September 2011
Strength of partnerships: a mixed methods approach to evaluation
Presented by:Date:
Geraldine Marsh
Virginia Lewis
Kate Silburn
Jenny Macmillan
Centre for Health Systems Development (CHSD) at the Australian Institute for Primary Care & Ageing (AIPCA) at La Trobe University has been evaluating initiatives that involve partnering at many levels.
Examples of the different types and levels of partnering initiatives we have evaluated include: Hospital Admission Risk Programs (HARP) Early Intervention for Chronic Disease projects Community Initiatives for National Suicide Prevention Strategy Two large-scale evaluations of statewide primary health system
development strategies in Australia: Primary Care Partnerships (Vic) and Connecting Healthcare in Communities (CHIC)
2
Role of Government
Governments have an important role in creating or enabling the foundations of effective systems for organised primary health care service delivery (Takach, Gauthier, Sims-Kastelein, & Kaye, 2010; World Health Organisation, 2008).
This requires deliberate and sophisticated policy processes that maximise the potential within complex systems and the capacity within these systems for innovation (McDaniel & Driebe, 2001; Swantson, et al., 2010; World Health Organisation, 2008).
Partnering and Primary Health Care Reform
Current reform is based on setting up (new) collaborative structures and relationships
Other geographically based formal and informal governance and organisational arrangements exist
Evaluative research is required to assess the emerging ML/LHN and their relationships to existing (or other developing) partnerships and the effects on PHC delivery and consumer outcomes
4
How do we assess value of partnering?
Evaluating partnering (collaboration, etc.) is important to demonstrate it adds value and is worth investment
Attributing the contribution of relationships between organisations to health service delivery and health outcomes is difficult
First step is being able to capture the nature of the partnership and describe its “effectiveness” in a systematic way
Measures
Literature on barriers and facilitators leads to an understanding of characteristics of functional partnerships, general consensus about what good partnerships look like. But how to capture systematically and usefully is difficult
Descriptive qualitative review Network mapping technique Questionnaires based on presence of characteristics
VicHealth Partnerships Analysis Tool Nuffield Partnership Assessment Tool (Hardy, Hudson and Waddington,
2003)
Commonly used questionnairesVicHealth Partnerships Analysis Tool
A tool for establishing, developing and maintaining productive partnerships
7 domains rated on a 5-point Agree/Disagree scale:1. Determining the need for the partnership2. Choosing partners3. Making sure partnerships work4. Planning collaborative action5. Implementing collaborative action6. Minimising the barriers to partnerships7. Reflecting on and continuing the partnership
Different possible methods of completion:Leaders present views for discussion; Individually completed; as a group activity
Nuffield Partnership Assessment Tool (Hardy, Hudson & Waddington, 2003)
Six domains rated on 4-point Agree-Disagree scale1. recognise and accept the need for partnership2. develop clarity and realism of purpose3. ensure commitment and ownership4. develop and maintain trust5. create clear and robust partnership working arrangements6. monitor, measure and learn
Completed individually with post-completion discussion
Potential issues
Each partner weights criteria for judging success differently (Toulemonde et al., 1998)
Sensitivity issues (e.g. Funnell, 2006; own experience leading to modified response options for original VicHealth tool)
Ratings tend to be “generous”
Discussion after completion of the tool is constrained by the ratings given
An alternative approach
Through extensive experience in evaluating partnerships CHSD has developed:
A mixed-method design for assessing partnerships in health systems
A generalised approach where focus of strategy/funding is on building capacity through inter-organisational (sector) processes
Procedure
Data collection involves a facilitated group interview ( partnership/governance group)
Questions are asked about the stage of development or progress: How well developed are each of the following in your Partnership Council?e.g. clear goals for PC
A facilitated discussion occurs
Two facilitators are present to allow post-completion review and moderation if required. Also enables accurate documentation of comments during discussion.
Questions (16) reflect domains represented in Nuffield and VicHealth
Includes references also to the underlying logic of the initiative/programIn the case of statewide primary health reform, some references to the health system within the catchment
Also collect a brief version of VicHealth Partnerships Analysis Tool (individually completed ) and a single overarching development question (consensus)
12
Response options relate to progress towards achievement of the characteristic:
We won’t be doing this Haven’t started yet Currently planning this Developing this This is reasonably well developed This is very well developed/ sustainable
No group consensus reached
Generality of the questions allows for flexibility and diversity in implementation of strategies, while facilitated discussion gathers specific information reflecting differences in local implementation the “why” is the same, the “where are we up to” is on the same scale, but
the “how” is different
Analysis of relationships between measures supports validity and reliability of the approach
15
Partnership Council ratings: “Clear goals for the PC ”
0
2
4
6
8
10
12
2008 (n=14) 2009 (n=14)
2011 (n=15)
Stages of Development
No.
of P
Cs
16
“Working so partners are involved in planning / setting priorities for collaborative action”
0
2
4
6
8
10
122008 (n=14)2009 (n=13)2011 (n=15)
Stages of Development
No. o
f PCs
Benefits of this Approach
Provides quantitative and qualitative data about the characteristics of the partnership
Immediate information is used by partnership to plan future actions – feedback supports strategic planning
Immediate information is used by funder/policy makers to identify areas where additional support and/or modification to strategy may be required
Qualitative information strengthens understanding and interpretation of quant data
17
Quantitative and qualitative data allows comparison of the characteristics of a partnership over time.
Contributes useful evidence to support development of the partnership and the policy/strategy
Data is about progress towards the strategic goal of increased system capacity through partnership.
Requires experienced and skilled facilitators – for the process of reaching consensus, and with ability to assess quality of evidence provided for ratings
Conclusion
The mixed method design can be applied to different settings to provide robust quantitative indicators with a rich level of interpretive information.
The data obtained through the facilitated group interview contributes useful evidence to support development of the partnership and the policy/strategy.
Associate Professor Virginia Lewis or Geraldine Marsh
Centre for Health Systems Development
Australian Institute for Primary Care & Ageing
La Trobe University
Email: [email protected] [email protected]
Contact:
Partnership Analysis Tool (Brief version – 20 items)
21
Determing the need for the partnership
Choosing partners
Making sure partnerships work
Planning and implementing collaborative action
Minimising the barriers to partnerships
Reflecting on and continuning the partnership
0
1
2
3
4
5
200820092011
1 = ‘Strongly disagree’2 = ‘Disagree’3 = ‘Not sure’4 = ‘Agree’ 5 = ‘Strongly agree’