research in see – volunteering and innovation
DESCRIPTION
Research in SEE – volunteering and innovation. Including results from: “T he impact of long-term youth voluntary service in Europe ”. Steve Powell , proMENTE social research, Sarajevo. O bjectives. Present briefly Evaluation of SEEYN workcamps AVSO/proMENTE review of impact studies - PowerPoint PPT PresentationTRANSCRIPT
Research in SEE – volunteering and innovation
Including results from: “The impact of long-term youth voluntary service in Europe”Steve Powell, proMENTE social research, Sarajevo
www.seeyn.org www.promente.org/v
Objectives
Present briefly – Evaluation of SEEYN workcamps– AVSO/proMENTE review of impact studies– Main findings– Main challenges in evaluating the impact of youth voluntary service
programs Intercultural learning: generic or dyadic? main results of researches on volunteerism done
by you. We want to ensure audience understand that volunteerism contribute to social inclusion and can be used to serve different purposes and goals
www.seeyn.org www.promente.org/v
Our review
presented in European Parliament June 2007: www.promente.org/avsoreview Funders included Global Service Institute / Center for Social
Development
www.seeyn.org www.promente.org/v
Studies reviewed
Over 300 documents analysed: published and unpublished research and evaluation studies
40 directly relevant to the impact of youth long-term voluntary service in Europe.
www.seeyn.org www.promente.org/v
European perspective
Long history and tradition of YVS Intensive, long-term placements, sometimes alone EC-funded EVS program Very heterogenous, country specific Surprisingly poor research base & tradition Poor responses from EVS national agencies
In spite of some encouraging results, overall the research conducted in Europe to date on the impact of voluntary service has had neither the methodological teeth nor the mandate to really test whether voluntary service works as advertised.
But whose job is it to demonstrate impact?
www.seeyn.org www.promente.org/v
The studies: sources
Some evaluation studies used many different sources.
All included information from volunteers and/or from implementing organisations
None included direct data from users / beneficiaries
The studies: data and designs
25 studies: qualitative data collection and analysis 30 studies: quantitative data and analysis Some studies used both kinds of data / analysis. Very few referred to any evaluation framework or toolkit Almost none considered drop-outs, attrition None used validated / composite scales or instruments
www.seeyn.org www.promente.org/v
Toolkits and frameworks
Model of voluntary activities and civic learning: (Mutz and Schwimmbeck 2005) VIVA (Gaskin 2003) Framework for organising service-related research (Perry and Imperial 2001) The functional approach (Snyder, Clary et al. 2000). AmeriCorps general theory of change model (Jastrzab, Giordono et al. 2004) Independent sector / UNV: Measuring volunteering: a practical toolkit (Dingle 2001) GSI: (Tang, Moore McBride et al. 2003) Council of Europe and European commission: International Voluntary Service T-kit (Amorim,
Constanzo et al. 2002) UNV: (Daniel, French & King 2006) IVR: (Institute for Volunteering Research 2004)
-Looking for - a meta-framework to present our findings- frameworks useful for individual studies None of the studies
made much use of toolkits or
frameworks
Great diversity in types and purposes
of study, approaches, focus on internal validity
IVR framework
www.seeyn.org www.promente.org/v
But ... Low comparability of studies + poor designs & reporting standards + lack of comparison, counterfactuals = Not possible to conduct a formal meta-analysis
Design Number of studies
Retrospective impressions
Nearly all; a few used narrative follow-ups
Comparison group 5
Before/After 2
Quasi-experiment 0
Dimension 2: the forms of capital (what kind of value is added?)
Physical Capital
Economic Capital
Human Capital
Social Capital
Organisations
Users
Volunteers
Dim
ensi
on 1
: Sta
keho
lder
(who
ben
efits
?)
Community
Decreased career indecision
active citizenship
Future volunteers
life-long learning
Improved organisational climate?
Reduced costs?
less wasted time at schools
better education & career choices
Value of services delivered
Personal growth
interest in social studies
broadened horizons
Social & communication skills …Tolerance / intercultural competence /
Improved staff skills?
job creation?
employability
generic skills and work experience
bridging social capital?
improved discipline?
Basic European identity
intention to continue to volunteer?
Potential to involve disadvantaged groups
employment?
Overview of findings
Many studies, at least some with good designs
Many studies, narrative / economic evidence
Many studies, weaker evidence
Anecdotal or mixed evidence
-Compare with results from today’s 3 studies?
www.seeyn.org www.promente.org/v
Main results
1. You get the impact you program for.– So voluntary service regularly produces those
kinds of impact for which voluntary service by its nature provides the input:
– Personal growth, independence– Career orientation, etc
-Compare with results from today’s 3 studies?
www.seeyn.org www.promente.org/v
Good news …
2. Voluntary service works for everyone!3. Often disadvantaged youth benefit more– Ceiling effect? tailor programs– Artefact? improve instruments
4. Differential effects matchmaking
-Compare with results from today’s 3 studies?
www.seeyn.org www.promente.org/v
Cold showerof numbers?
5. Even more objective methods do indeed reveal some significant benefits
6. But always much less encouraging than retrospective reports
7. And there are some disappointments– .. Takes courage
-Compare with results from today’s 3 studies?
Evaluation functions
Learning & management
Control & accountability
Demonstrating impact
Individual projects & programs
... More room for ad-hoc, emic approaches
... Individual studies? RBM? Or (partial) adherence to model?
Policy level ditto ... Occasional, high quality studies standards & models
Studies varied greatly in their
functions
www.seeyn.org www.promente.org/v
Main challenges: mainstream YVS impact evaluation culture
Subjective– The whole gamut of biases– Vested interests
Stronger internal validity and very varied approaches at program level Low motivation to evaluate Idealism & the deep fear of the cold shower Bad influence of RBM & LFA
– (every implementation has to demonstrate hard-to-measure impact) Non-science: lack of counterfactuals, even before/after comparison Pre-science: lack of
– comparability/generalisibility– seminal papers– accepted research paradigm & tools
Fixable?
www.seeyn.org www.promente.org/v
Final suggestions Impact measurement toolkit, not a one-size-fits-alls “framework”– Wide selection of free, documented & validated
impact measurement tools and methods (e.g. leverage WVS) Don’t require “evaluations” at project level! ... ... Fewer, higher-quality impact studies at sector / country level– ... which can do some of the work for program-level studies
• Validate the tools• Standardise some program elements• Answer research questions
– Typical 1-year change in non-volunteers on key measures like job self-efficacy– Is better improvement on key outcomes in disadvantaged youth an artefact?– Cost benefits of reducing career indecision?– ....
