rmbc cmf theory of change web version · title: microsoft powerpoint -...
TRANSCRIPT
Rotherham Building Stronger Communities
Draft ‘Theory of Change’ for the evaluation Steve Connelly
University of Sheffield1st February 2018
Introduction
This slide pack sets out a draft ‘theory of change’ for the CMF-funded projects within RBSC. It is intended to • be the basis for a structured approach to the
evaluation of the programme• prompt discussion about what can and should be
evaluated, and how• lead to better understanding of how the projects might
work, where there might be synergies and overlaps and potential conflicts, and identify previously unpredicted outcomes (good and bad)
• identify areas of lack of clarity for further discussion and development
Structure
• key points and outstanding questions and issues
• logic models (LM), theories of change (ToC), and the context/mechanism/outcome approach
• BSC’s 9 types of mechanism, and relationship to projects and outcomes
• an example of a ToC for one mechanism• the full ToC• an example of individual project logic models
key points• it’s very complex – even ‘simplified’ down to 9 linear ways
of having effects (‘mechanisms’)• most mechanisms have multiple outcomes, so we have an
‘attribution problem’ – which projects caused what?• the project deliverables are focused very much on
activities, less on the processes by which they will have effects (capacity change (8), behaviour change (10), direct benefits (7))
• of the 16 deliverables related to wider benefits, only 5 are attributed to projects
• the (understandable) project focus on ‘doing stuff’ with communities leaves out tackling contextual factors in both communities and state/3rd sector bodies: how do we ensure support for enduring change both ‘culturally’ and in material terms?
outstanding questions/issues
• how can the complexity of the project be presented in ways which enable informed partner input?
• we need more measures of if/how activities are having effects, and how these lead to benefits
• what (if any) evidence do we currently have to support the causal links and assumptions?
• what differences might there be between groups in terms of how relevant/accurate the assumptions are? and so of how effective the intervention is?
• what risks are associated with the assumptions? what measures can be taken to mitigate them?
• how do we examine each activity for its sustainability?
Logic models
logic models set out the logic of an intervention: how it is supposed to have effects
Adapted from Mayne, J (2015) ‘Useful Theory of Change Models’, Canadian Journal of Program Evaluation, 30 (2) 119-142
Activities/ outputs
Capacity change
Behaviour change
Direct benefits
Wider benefits
Logic models + assumptions become theories of change
• a full ‘theory of change’ (ToC) includes the ways in which the logic model stages are assumed to connect.
• these assumptions also point to conditions which have to be present if the LM is to be made real (i.e. if the intervention is to work) and suggest alternative outcomes
Assumptions about reach, how capacity is changed,
and conditions required
Activities/ outputs
Capacity change
Behaviour change
Direct benefits
Wider benefits
Assumptions about how
capacities lead to behaviour change, and conditions required
Assumptions about how behaviour
change benefits the
individual
Assumptions about how
direct benefits lead
to wider social
benefits
Assumptions about the
basic purpose and function of the activity
mechanism + context → outcome
• evaluation theorists also use a language of ‘mechanism’ for the process which leads from activity to benefits, and the idea that mechanisms act in contexts to lead to outcomes: the same mechanism in a different context might lead to different outcomes
• here ‘mechanism’ means the first three stages of the LM...
...and ‘outcomes’ are the benefits
Activities/ outputs
Capacity change
Behaviour change
Direct benefits
Wider benefits
9 types of mechanism identifiable across the projects
Working with key individuals
Providing advice
Training/education
Getting individuals together across communities
Environmental projects
Financial support for community groups
Targeted state support to families and young people
Increasing enforcement activity
Working with governance organisations (state and VCF)
9 types of mechanism, 8 partners, 24 projects
Working with key individuals
Providing advice
Training/education
Getting individuals together across communities
Environmental projects
Financial support for community groups
Targeted state support to families and young people
Increasing enforcement activity
Working with governance organisations (state and VCF)
RMBC: ACE
RMBC: CYPS
RMBC: RE
Clifton Learning Partnership
REMA
RUCST
Premier Learning
Kimberworth Park Community P’ship
2
8
3
19
5
10
1
3
9
20
6
11
4
7
14
15
16
22
23
3 12
13
14
15
12
14
15
16
18
21
mechanisms in detail (1)
Activities/ outputs
Capacity change
Behaviour change
Working with key
individuals
Providing advice
Training/education
Getting individuals together
across communities
Street champions
Community navigators
Good neighbour project
Increased engagement skills, confidence,
knowledge to - provide help to others
- engage with the state/VCF
- mediate disputes
Engagement with others in the community
Reporting problems to relevant governance organisations
Drawing up a ‘Residents’ Charter’
Advice services Increased knowledge of rights, services available
More, faster, earlier, appropriate accessing of services
ESOL, and other skills/knowledge taught
along with ESOL
Better knowledge of English
Better understanding of rights/responsibilities
Other knowledge/skills (with vocational etc.
relevance)
More engagement with service providers
More exercising of democratic rights
More socially responsible action within communities
Move into employment
ESOL, Citizenship Project, BSC Forum,
festivals/events, sport, Love is Louder events
Better understanding, trust, respect for different Others
Less antagonistic behavioursEnforcement of norms of
tolerance within communities (e.g. stopping/reporting hate
crime)Self-organised inter-community
collaborative working Better school attendance
mechanisms in detail (2)
Environmental projects
Financial support for community
groups
Targeted state support to families and young people
Increasing enforcement
activity
Working with governance
organisations
Activities/ outputs
Capacity change
Behaviour change
CLP environmental projects
Provided with skills and opportunity to deliver projects More respect amongst participants and others for the immediate environment
Direct improvements: litter cleared, allotments, plantingLess littering in futureMore food growing Better school attendance
Giving small grants to community
groups/organisations Providing support to
groups to bid
miscellaneous miscellaneous
Outreach/street working with young people
Targeted bespoke family support
YP more confident and self-aware as citizensBetter (state) understanding of YPs’ issues
Reduced ASB (including drugs)Collective activities Participation in youth forumIncrease in education participation ratesReduction in family abuse
Reducing CTax fraudEnforcing housing
standardsEnforcing environmental regulations - fly tipping
Understanding of risks and penalties for non-compliance/lawbreakingStaff and IT (surveillance, reporting) increased
Cessation of illegal activityCompliance with rules
BSC Forum?
Better understanding of communitiesTrust, understanding, skills to work in partnership Better knowledge of possible funding/resource streams
More appropriate service provisionMore effective partnership working: reduced duplication, signposting between organisations, new joint actionsMore external resources successfully bid for
mechanisms and outcomes
Working with key individuals
Providing advice
Training/education
Getting individuals together across communities
Environmental projects
Financial support for community groups
Targeted state support to families and young people
Increasing enforcement activity
Working with governance organisations (state and VCF)
Direct benefits
Quicker settling in
More effective resolution of service problems
More harmonious community life (less involvement in racial abuse etc., disputes resolved)
Satisfaction/wellbeing from social engagement/participation
More confident and able to exercise rights/responsibilities
(Better) employed/higher income
Living in better physical environment (greenery, litter, fly tipping…)
More secure tenancy in better housing
More harmonious personal lives (within families, at school, less friction with authorities)
Mechanisms
mechanisms and outcomes
Working with key individuals
Providing advice
Training/education
Getting individuals together across communities
Environmental projects
Financial support for community groups
Targeted state support to families and young people
Increasing enforcement activity
Working with governance organisations (state and VCF)
Direct benefits
Quicker settling in
More effective resolution of service problems
More harmonious community life (less involvement in racial abuse etc., disputes resolved)
Satisfaction/wellbeing from social engagement/participation
More confident and able to exercise rights/responsibilities
(Better) employed/higher income
Living in better physical environment (greenery, litter, fly tipping…)
More secure tenancy in better housing
More harmonious personal lives (within families, at school, less friction with authorities)
Mechanisms
mechanisms and outcomes
Working with key individuals
Providing advice
Training/education
Getting individuals together across communities
Environmental projects
Financial support for community groups
Targeted state support to families and young people
Increasing enforcement activity
Working with governance organisations (state and VCF)
Direct benefits
Quicker settling in
More effective resolution of service problems
More harmonious community life (less involvement in racial abuse etc., disputes resolved)
Satisfaction/wellbeing from social engagement/participation
More confident and able to exercise rights/responsibilities
(Better) employed/higher income
Living in better physical environment (greenery, litter, fly tipping…)
More secure tenancy in better housing
More harmonious personal lives (within families, at school, less friction with authorities)
Mechanisms
direct and wider benefits
Wider benefitsDirect benefits
Quicker settling in
More effective resolution of service problems
More harmonious community life (less involvement in racial abuse etc., disputes resolved)
Satisfaction/wellbeing from social engagement/participation
More confident and able to exercise rights/responsibilities
(Better) employed/higher income
Living in better physical environment (greenery, litter, fly tipping…)
More secure tenancy in better housing
More harmonious personal lives (within families, at school, less friction with authorities)
More appropriate service provision
More appropriate calls on services
More participation in ‘governance’ broadly
Less hate crime and other ASB
More meaningful interactions between different groups
Better landlord/tenants relationships
Better neighbourhood environment
Better housing (physical, security)
Wealthier community
Higher (mental/physical/ emotional) well-being
Positive changes in state (etc.)/resident
behaviour
Positive changes in resident/resident
behaviour and relationships
Improved conditions for (deprived) communities
direct and wider benefits
Wider benefitsDirect benefits
Quicker settling in
More effective resolution of service problems
More harmonious community life (less involvement in racial abuse etc., disputes resolved)
Satisfaction/wellbeing from social engagement/participation
More confident and able to exercise rights/responsibilities
(Better) employed/higher income
Living in better physical environment (greenery, litter, fly tipping…)
More secure tenancy in better housing
More harmonious personal lives (within families, at school, less friction with authorities)
More appropriate service provision
More appropriate calls on services
More participation in ‘governance’ broadly
Less hate crime and other ASB
More meaningful interactions between different groups
Better landlord/tenants relationships
Better neighbourhood environment
Better housing (physical, security)
Wealthier community
Higher (mental/physical/ emotional) well-being
Positive changes in state (etc.)/resident
behaviour
Positive changes in resident/resident
behaviour and relationships
Improved conditions for (deprived) communities
theory of change for one mechanism
• Getting individuals together across communities
Antagonism between communities, and
towards the state for perceived
biases/failings, comes from a lack of
understanding of, and empathy with,
different Others, and direct contact will
address this (rather than worsen it)
Activities: ESOL, Citizenship Project,
BSC Forum, festivals/events,
sport, Love is Louder events
Capacity changes: Better
understanding, trust, respect for different Others
Behaviour changes: Less antagonistic behavioursEnforcement of norms of tolerance within communities (e.g. stopping/reporting hate crime)Self-organised inter-community collaborative working Better school attendance
Direct benefits
Wider benefits
People who engage are those whose positions need changing Contact provides opportunities for effective, meaningful dialogue, and learning about the Other – and not for increased antagonism/reinforcing of prejudices.Sufficient, competent staff are involved, who can (at minimum) effectively deal with tension/friction, and are able to facilitate effective (potentially very difficult) interactions
Individuals are sufficiently confident to act on new understandings within their cultural context, and robust enough for this to be sustained Opportunities and support are available for further engagement Local and national policy, the media and wider societal discourses, do not provide overriding counter-messages
Behaviour changes are enduringSkills are successfully deployed (personal and mediation etc.) Communities respond positively to changed behaviours
Mechanisms Activities Capacity change Behaviour changeWorking with key individuals Street champions
Community navigatorsGood neighbour project
Increased engagement skills, confidence, knowledge to - provide help to others- engage with the state/VCF- mediate disputes
Engagement with others in the community Reporting problems to relevant governance organisations Drawing up a ‘Residents’ Charter’
Assumptions Tacking problems requires communication (at minimum) between state and residents, and key individuals can provide routes into the community which the state (and VCF?) cannot achieve themselves, due to problems of trust, legitimacy, language, resource capacity etc.
Suitable individuals can be identified, and will keep with the programmeProvider organisations are trusted enough for relevant people to engage with them Programmes will deliver the required skills and knowledge
Participants will be willing and able to actParticipants will share the programme goals and act appropriately Participants will have the networks and legitimacy to actCultural norms support increasing engagement with the state Service providers will respond and reinforce behaviour
Providing advice Advice services (REMA and CLP) Increased knowledge of rights, services available More, faster, earlier, appropriate accessing of servicesAssumptions ignorance is a problem - people would
value services/rights but fail to take them up or act on their rights because they don’ know what is available, how to access them etc. or what their rights are.
Those in need will know about the services- and have the confidence and ability (mobility, language) to access the serviceAppropriate and accurate advice will be available, from people with the skills/attitudes/knowledge to engage with the target population
People will have the confidence and resources to act on the knowledgeCultural norms support increasing engagement with the state Appropriate services will in fact be available and accessible and able to meet any increased demand
Training/education ESOL, and other skills/knowledge taught along with ESOL
Better knowledge of EnglishBetter understanding of rights/responsibilities Other knowledge/skills (with vocational etc. Relevance)
More engagement with service providersMore exercising of democratic rightsMore socially responsible action within communitiesMove into employment
Assumptions People lack knowledge and skills needed to play a full part in society -and they see education/training as a way of addressing this
Those in need will know about what is available- and have the confidence and ability (mobility, language) to access the servicePeople will be able to attend regularly enough to learnSufficient high quality training, appropriate to users’ needs, is available
Education/knowledge will empower peopleCultural norms support increasing engagement with the state Service providers are willing/able to engage and meet increased demandGovernance organisations are willing/able to engage with democratic expression
Getting individuals together across communities
ESOL, Citizenship Project, BSC Forum, festivals/events, sport, Love is Louder events
Better understanding, trust, respect for different Others Less antagonistic behavioursEnforcement of norms of tolerance within communities (e.g. stopping/reporting hate crime)Self-organised inter-community collaborative working Better school attendance
Assumptions Antagonism between communities, and towards the state for perceived biases/failings, comes from a lack of understanding of, and empathy with, different Others, and direct contact will address this (rather than worsen it)
People who engage are those whose positions need changing Contact provides opportunities for effective, meaningful dialogue, and learning about the Other –and not for increased antagonism/reinforcing of prejudicesSufficient, competent staff are involved, who can (at minimum) effectively deal with tension/friction, and are able to facilitate effective (potentially very difficult) interactions
Individuals are sufficiently confident to act on new understandings within their cultural context, and robust enough for this to be sustained Opportunities and support are available for further engagement Local and national policy, the media and wider societal discourses, do not provide overriding counter-messages
the full ToC: mechanisms and assumptions
Mechanisms Activities Capacity change Behaviour changeEnvironmental projects CLP environmental projects Provided with skills and opportunity to
deliver projects More respect amongst participants and others for the immediate environment
Direct improvements: litter cleared, allotments, plantingLess littering in futureMore food growing Better school attendance
Assumptions Antagonism and dissatisfaction with neighbourhoods related to poor environmental condition, and social and physical improvements can be achieved if people improve the environment themselves
People with enthusiasm can be recruited and retained
Project outputs can be maintained (and not vandalised) – either by the state or by the community Improved environment encourages more environment-friendly behaviours
Working with governance organisations (state and VCF)
BSC Forum? Better understanding of communitiesTrust, understanding, skills to work in partnership Better knowledge of possible funding/resource streams
More appropriate service provisionsMore effective partnership working: reduced duplication, signposting between organisations, new joint actionsMore external resources successfully bid for
Assumptions Service provision is currently weakened by governance issues, in particular around integrated/partnership working, and (lack of ) understanding of relevant communities (in particularly Roma)
Service providers – as organisations and individuals – are able and willing to learnGovernance organisations are willing to work together
Service providers are adequately resourced to act on new understanding Governance organisations are able to work together (adequate resources/support available)
Increasing enforcement activity
Reducing CTax fraudEnforcing housing standardsEnforcing environmental regulations - fly tipping
Understanding of risks and penalties for non-compliance/lawbreakingStaff and IT (surveillance, reporting) increased
Cessation of illegal activityCompliance with rules
Assumptions Violations of environment/housing regulations play a significant role in creating antagonism between communities, and stricter enforcement can be done effectively without antagonising communities/undermining trust in the state
People beyond those directly affected know about enforcement action and the risk of itPeople ‘read’ enforcement in terms of upholding law, not as oppression Intelligence – from IT and staff/community reporting – can be effectively and swiftly acted upon
Calculations of costs/benefits of illegal/no-compliant activity favour cessation and compliance (may be affected by perceptions of risksProvision of services to enable compliance (e.g. of waste disposal, support for housing repairs)
Targeted state support to families and young people
Outreach/street working with young peopleTargeted bespoke family support
Better (state) understanding of YPs’ issues Reduced ASBCollective activities Participation in youth forumIncrease in education participation ratesReduction in family abuse(Reduction in drug use?)
AssumptionsFinancial support for community organisations/groups
Giving small grants to community groups/organisations Providing support to groups to bid
Assumptions Appropriate groups can be reachedSmall financial barriers are important in preventing groups achieving aimsLack of capacity/knowledge/confidence in bidding is a significant barrier
the full ToC: outcomes and assumptionsDirect benefits(Better) employed/higher income
Assumptions: Language skills and confidence are barriers to employmentRelevant and accessible jobs exist
Better physical environment (greenery, litter, fly tipping…)Assumptions: Sufficient numbers of people participate, and continue to participate/act
Other members of community and service providers support actions
More effective resolution of service problems
Assumptions: Service providers are willing/able to respond to demandMore harmonious community life (less involvement in racial abuse etc., disputes resolved)
Assumptions: Behaviour changes are enduringSkills are successfully deployed (personal and mediation etc.)
Communities respond positively to changed behaviours
More secure tenancy in better housingAssumptions: Landlords are able/willing to make investments
Tenants are able/willing to be ‘good tenants’ – longer lets, less damage etc.
Quicker settling inAssumptions: New arrivals are reached, and are willing/able to respond
Service providers are willing/able to respond to demand
Satisfaction/wellbeing from social engagement/participationAssumptions: Engagement is in fact positive (and not stressful etc.)
Benefits are significant in relation to other causes of ill-being
More confident and able to exercise rights/responsibilitiesAssumptions: Participation and learning are sufficient to enable active engagement
Service providers, governance organisations, other members of community are receptiveOpportunities for engagement exist and are accessible
More harmonious personal lives (within families, at school, less friction with authorities)Assumptions: Service providers, governance organisations, other members of community are receptive
and supportive.