Download - Big lottery fund evidence briefing
Improving Futures Evaluation and Learning
Evidence Briefing PresentationLaurie Day and James Ronicle, Ecorys UK
21st January 2014
Part 1:
Evaluation Overview & Methodology
Laurie Day, Evaluation Director
Key Indicators Improving Futures
• Launched by the Big Lottery Fund in March 2011
• Total £26 million grant funding distributed between 26 pilot projects across the UK
• Aiming to test different approaches to improve outcomes for children in families with multiple and complex needs
• Requirements: – led by VCOs partnership with statutory services– include mechanisms to engage 'hardest to reach' families; and. – oldest child 5-10 years at entry stage
Measuring Impact Aims of the Evaluation
Evaluation aims 1.To measure the effectiveness, impact and outcomes of the 26 projects; and the programme as a whole2.To support the projects with measuring progress 3.To capture and share learning across the programme, and to disseminate to policymakers & practitioners across the UK
Scope •Consortium led by Ecorys, with Ipsos MORI, Family Lives and University of Nottingham •Five year evaluation (Oct 2011 - Dec 2016)•Evaluation and learning
Health and wellbeing
Emotional and behavioural
development
Educational outcomes
Safeguarding and social care
Crime and ASB
Parenting skills and confidence
Health and wellbeing
Educational outcomes
Safeguarding and social care
Crime and ASB
Family functioning and relationships
Financial wellbeing and
security
Secure and safe environment
Social networks and belonging
Strategic influence over UK policyLeverage over external resourcesSynergy with other programmes
Knowledge transferReplication of effective practice
Net improvements to children’s life chancesNet risk reduction for families with complex needs
Net attributable savings
Programme Effectiveness
Programme-levelEffectiveness of programme design,
development and implementation
Project-levelEffectiveness of governance;
partnerships; strategy and planning; user involvement and delivery
Quality of provision Sustainability
Children Adults Families
Evaluation FrameworkProgramme Impacts
Programme Outcomes
Strategic Benefits
Measuring Impact Methodology
Programme-Level Evaluation
•Literature review
•Monitoring data collection (IFMIS): secure online system (practitioner-led)
•Longitudinal survey of families: baseline; +12 mths, and +24 mths
•Stakeholder telephone survey: key local stakeholders (2013 & 15)
•Case study research: mix of longitudinal (n=6) & snapshot (n=14)
•Family Advisory Panel
Project-level evaluation
•Co-production of local evaluation plans: support with identifying project outcome measures & data sources
•Bi-annual monitoring against local grant milestones, and qualitative feedback on lessons learned
•Support for self-evaluation: advice on methods, tools and training
•Enhanced evaluation activities: CBA case studies; impact case studies; participatory research case studies
Measuring Impact Key considerations •Need to gather comparable data: measuring changes to families’ status over time •26 local projects, with different local data systems
Format Secure online databaseData entry by project workers, based upon data recorded within service plan (or equivalent) Three mandatory data entry points: entry, exit, + 6mths post-exit (with optional interim measure)
IFMIS: Overview
Key Indicators Key Indicators
• Child, adult and family indicators, based on a review of the research literature, plus socio-demographic data
• Problems and strengths; hard and softer measures
• Three indicator types : dynamic (subject to change), event-based (time-specific), and status (interpretive only)
• Record type and amount (hours) of support received
Measuring Impact Creating a family record – example
Measuring Impact Measuring Impact Adding and updating indicators – example Impact
Measuring Impact Measuring Impact and Outcomes
1. Desk research to develop key indicators (risk factors & strengths)
2. Embed within monitoring framework (IFMIS)
3. Measure short-term change over time (monitoring data)
4. Measure longer-term outcomes, up to +2yrs (survey)
5. Test predictive ‘strength’ of indicators (compare survey & MI)
6. Scale-up to estimate programme-wide effects
Measuring Impact
• Need to establish ‘additionality’ of the programme
• Counterfactual analysis: explore alternative scenario (no intervention). Two approaches:
1. Self-reporting of impacts: all projects (exit questionnaire and via Ipsos MORI survey)
2. Quasi-experimental design: 4-5 impact case study projects: compare outcomes between participants and matched comparison group (“difference-in-differences”)
Counterfactual
Evaluation Activities Completed to Date
1. IFMIS system live and fully embedded
2. Bespoke evaluation plans in place for all 26 projects
3. Baseline survey of families underway
4. First local stakeholder survey underway
5. Six ‘trailblazer’ case study visits completed (England, Wales, and Scotland)
6. Participatory action research training delivered
Measuring Impact Learning and dissemination strand
Policy Makers, Practitioners, Wider StakeholdersImproving
Futures Projects
Good Practice Publications
Webinars
Policy Workshops
Face to Face Learning Events
Project Exchanges
Shared Learning Environment
Electronic Newsletters
Press Releases
Policy Briefings
Policy Makers, Practitioners, Wider
Stakeholders
Measuring Impact Learning activities completed to date
Internal1.National launch event for the programme 2.Secure online portal for projects3.Project networking events and webinars 4.Newsletter
External 1.Launch of evaluation web-site: http://www.improvingfutures.org/ 2.Social media activities 3.Policy round table session
Measuring Impact
Part 2:
Key Findings from Year 1 of the Evaluation
James Ronicle, Evaluation Project Manager
Introduction
- Progress in Establishing Projects
- Supporting Families
- Who are the Improving Futures Families?
- Early Outcomes for Families
- Conclusions and Next Steps
- Implications and Lessons Learnt for Future BIG Programmes
• Focus to date has been on building local partnerships and delivery teams.
• Projects still in early stages of delivery & models are evolving
• Number of families supported by projects are lower than anticipated. However, referral numbers picked-up considerably during 2013.
• Projects face more challenging financial climate than when launched.
• Projects faced barriers to partnership working due to heightened sense of competition & widespread restructuring
• Despite this, projects building strong local partnerships: - Engaging public services - Collaborations between VCSs - Engaging ‘micro’ VCSs
• Consequently, projects sharing best practice & early signs becoming embedded within local structures for children & families.
Progress in Establishing Projects
Supporting Families: Examples of Improving Futures ProjectsThe Neighbourhood Alliance, Sunderland•Led by Foundation of Light, Sunderland Football Club•Developed ‘Neighbourhood Menu’ of 45 ‘micro- VCSs’ to support families
BIG Manchester•Created co-located team to tackle ‘toxic trio’ of substance misuse, mental health & domestic violence•Provides three areas of support:
• Holistic family support through Key Worker• Peer and community engagement• Spot purchasing of specialist support
Family Pathways, Lewisham•Focuses on children demonstrating destructive or anti-social behaviour•Runs ‘Roots of Empathy’; classroom-based programme where children spend time with babies to strengthen emotional competence• Delivers training for
practitioners on promoting attachment between parents & children
Tyne Gateway, N & S Tyneside•Recruits Family Entrepreneurs as ‘barefoot professionals’:
• Outreach• Support in assessing family needs• Support in delivering services• Direct support to families
Supporting Families:Examples of Projects
Supporting Families:Identification and Engagement
• Primary schools have provided a hub for many projects (59% of families engaged through schools)
• Challenges around reaching out to ‘harder to engage’ families who do not traditionally participate
• Previous negative experiences of contact with services having an impact on willingness to engage
• Some challenges around 5-10 age criteria
• Greater number of families with complex needs than anticipated, in some projects
Supporting Families:Packages of Support
• Replicating many aspects of good practice :• ‘Key Worker’ model• Holistic support packages• Evidence-based parenting programmes:
• Incredible Years, Triple P, Roots of Empathy
• Innovative finance mechanisms:
• Family budgets• Spot purchasing
• Peer support
• However, not all projects providing ‘holistic’ support
• Key success factors:
• ‘Out of hours’ provision• Frequent Key Worker contact• Hands-on approach with referrals to other services
Measuring Impact Who are the Improving Futures Families?Demographics
• Free School Meals significantly above national averages (77%)
• Minority ethnic background – above UK averages (at 31%). Variation from previous family programmes (FIPs: 88% white, and FPs: 77% White-British)
• High proportion of lone parent families – approaching two thirds, at entry stage (62%).
• Predominately female adult profile (75%)
Measuring Impact Who Are the Improving Futures Families?Family Risk Factors
Figure 1: Family risk factors affecting 10% or more of the cohort
Base: 891 families
• ‘Upper-middle’ in continuum of need. More prevalent than general population; less prevalent than other intensive intervention programmes.
Measuring Impact Who Are the Improving Futures Families?Adult indicators
Risks (-)
•Parenting difficulties the most prevalent issue: parenting anxiety or frustration (56%), problems with discipline / boundaries (49%)
•Well over one third with suspected or reported stress or anxiety (41% - twice the population average)
•Far lower prevalence of adult ASB or crime indicators than some other programmes (2.1% ASB, compared 79% for FIPs).
Strengths (+) •Over half of adults within the cohort (51%) demonstrating awareness of how to keep children safe from harm, and around one third setting appropriate boundaries (34%)
•Fair levels of face-to-face contact with school staff and positive relationships, ranging from under half for all adults (45%) to just over half for lone parents (52%)
Measuring Impact Who Are the Improving Futures Families?Child indicators
Risks (-)
•Child behavioural problems by far the most prevalent risk, when low-level (30%) and persistent disruptive behaviour (16%) are combined
•One in five children with suspected or reported stress or anxiety (22%); mirroring similar problems in adults within the cohort, and one in ten under-achieving at school
•Lower propensity for project workers to report risk factors for children; could indicate recording difficulties
Strengths (+) •Most children adopt some aspects of a healthy lifestyle: well over half (61.5%) attend regular medical appointments; a fair proportion participate in play (37%), and exercise or sport activities (27%)
•Over one third of children have supportive peer friendships at school (43%), whilst just over one quarter have regular contact with friends outside school (27%).
Measuring Impact Who Are the Improving Futures Families?Towards some typologies…
• Factor analysis of IFMIS data (in SPSS) – clusters of variables that are strongly correlated with each other but relatively uncorrelated with variables in other clusters
• Iterative approach to find the most robust and realistic groupings
• Eight typologies identified on this basis...
Measuring Impact Who Are the Improving Futures Families?Towards some typologies (2)
• Typology 1 - Families with strong structures and behaviours
• Typology 2 – Low skilled families with financial and housing difficulties
• Typology 3 – Families with significant discipline and behavioural issues
• Typology 4 – Transient families with domestic abuse, mental health and substance misuse
• Typology 5 – Families engaged in their community
• Typology 6 – Families known to the police for minor disorder
• Typology 7 – Socially excluded families with health problems
• Typology 8 – Families known to the police for more serious incidents
Early Outcomes for Families
1. Projects able to achieve ‘quick wins’ for families:
1.Resolving financial problems
2. Improving housing circumstances
2. Parents reporting improved well-being:
1.Confidence & self-esteem
2.Reduced Anxiety
3.Parenting capacity
3. Improved behaviour and attendance at school
Evaluation Conclusions•Improving Futures is becoming integrated into wider local support infrastructure for children and families
•The programme is strengthening links between statutory and VCS organisations
•Projects showcased diverse range of practices for working with families at different levels of need. Some are replicating good practice, others developing good practice
•Too early to offer robust conclusions about impact of programme, but positive early signs that projects making different to families’ lives.
Areas for Development•Reaching families not in contact with mainstream services
•Taking stock of ‘whole family’ approaches
Conclusions and Next Steps
Evaluation design
•Need to strike a balance between a) freedom for grant-holders to innovate, and b) setting parameters for measuring impact
Programme design
•Value of statutory (DCS) endorsement of VCS bids•Development phase critical for building multi-agency partnerships (…but make next stages conditional on early success?)
Key areas of added value / potential legacy…
•Understanding dynamics of change for families over time•What works with the 5-10 age group, and VCS-school links •New forms of service commissioning (esp. neighbourhood level)•Models of engagement and support for BME families
Implications / Lessons Learned for Future Programmes