![Page 1: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/1.jpg)
Telling the Difference between Snake Oil and Good Oil; Evaluation Capacity Building in the public sector
Bronte Koop, Principal Consultant, Strategy and Evaluation
John Collins, Manager, Program Evaluation
![Page 2: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/2.jpg)
COAG Reform Agenda: Human Capital Development
• National Partnerships
• Literacy and Numeracy
• Teacher Quality
• Low SES communities
![Page 3: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/3.jpg)
• Not getting what you expected from an evaluation
![Page 4: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/4.jpg)
What do we mean by Snake Oil
• Staff as passive recipients
• Evaluation questions not addressed
• Use lots of words
• Recommendations to continue the program and evaluate it again
• Lack of evidence based recommendations for improvement
• No discussion of data or design limitations
• No discussion of alternative explanations for program effect
• Evaluators reluctant to address client concerns about clarity or focus
![Page 5: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/5.jpg)
![Page 6: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/6.jpg)
• Useful information to guide decision making
![Page 7: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/7.jpg)
So what’s the Good Oil
• Staff critical participants
• Provides answers to specific questions
• Balances qualitative with quantitative data
• Reports are just sufficient in length
• Conclusions follow from the evidence
• Has logical recommendations
• Data or design limitations are discussed
• Alternative explanations for program effect are discussed
![Page 8: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/8.jpg)
How do we get there?
teamwork
commitment
learning proactivity
![Page 9: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/9.jpg)
What are we trying to change?
Attitudes and values Knowledge and skills
Organisational behaviour
![Page 10: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/10.jpg)
Overcoming your fears is the first step
![Page 11: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/11.jpg)
Principles for evidence based program management
• Demand evidence
• Examine the logic
• Treat the program as an unfinished prototype
• Embrace an attitude of wisdom
Jeffrey Pfeffer and Robert Sutton on evidence based management
![Page 12: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/12.jpg)
1. Executive leadership2. Consulting Services3. Resources4. Professional development5. Policy and operations6. Networks7. Strategic Oversight
Where should be put our efforts?
CREATING
CHANGE
![Page 13: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/13.jpg)
• Locate near the top
• Recruit the knowledge, skills and motivation
• Build credibility
• Provide an ‘evaluation perspective’
Consulting Services
![Page 14: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/14.jpg)
Resources
If you build it, they will come
![Page 15: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/15.jpg)
What have others done?
![Page 16: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/16.jpg)
Accessible Resources
![Page 17: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/17.jpg)
Practical Resources
![Page 18: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/18.jpg)
Practical Resources
Provide a starting point
![Page 19: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/19.jpg)
Project Management
![Page 20: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/20.jpg)
Stage 1Initiation
Stage 2Planning
Stage 3Execution
Stage 4Transition & Closure
· Project Proposal
· Appoint project sponsor and board
· Appoint project manager· Investigate similar projects· Discuss scope and goals· Discuss evaluative activities· Gain approval to plan
Project Management Templates
Program Evaluation Templates
Program and Project Management Tasks
· Logic Model· Project Plan· Budget· Aboriginal Impact· Gateway· Evaluation plan
· Status reports· Progress monitoring· Change of scope· Transition plan
· Project closure report· Evaluation report
· Evaluation plan· Good practice checklist· Logic model and performance
measures· Focus on expected changes· Stakeholder perspectives· Resource assessment· Developing questions· Choosing an evaluation design· Data collection and sampling
· Participant consent· Engaging an external evaluator· Evaluation brief· Evaluation work plan
· Final report· Quality assurance checklist· Reflecting on management
effectiveness
Guide to project and program management stages
· Define goals and outcomes· Identify evaluation questions· Plan activities· Schedule activities· Plan evaluation· Gain approval to implement· Identify sources of evidence· Scope data requirements
· Conduct team meetings· Monitor implementation· Liaise with governance groups· Conduct implementation
evaluation· Report on progress· Modify scope where needed· Gain approval for change
· Implement transition· Conduct post project review· Respond to evaluation findings· Disseminate evaluation findings· Finalize activities
Evaluation & planning
![Page 21: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/21.jpg)
Professional Development
Time to stretch your thinking
Induction
Customized training to needs
![Page 22: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/22.jpg)
Procurement:Expression of Interest
Two part contracts: plan + do
Executive summariesInterviewsHard questions
Policy and Operations
Think outside the square
![Page 23: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/23.jpg)
Network, network, network
![Page 24: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/24.jpg)
Strategic objective 2011
“We will foster a culture of evaluation and review to
better direct our services.”
![Page 25: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/25.jpg)
Strategic objective 2011
“We will foster a culture of evaluation and review to
better direct our services.”
![Page 26: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/26.jpg)
Strategic Plan 2012-2016
• We take seriously our responsibility to invest public funds effectively and efficiently.
• Our funding decisions will be guided by the potential of our policies and programs to improve learning outcomes . . .
![Page 27: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/27.jpg)
Strategic Plan 2012-2016
• We take seriously our responsibility to invest public funds effectively and efficiently.
• Our funding decisions will be guided by the potential of our policies and programs to improve learning outcomes . . . current investments will be evaluated and we will reinvest where appropriate.
![Page 28: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/28.jpg)
Strategic Plan 2012-2016
• We take seriously our responsibility to invest public funds effectively and efficiently.
• Our funding decisions will be guided by the potential of our policies and programs to improve learning outcomes . . . current investments will be evaluated and we will reinvest where appropriate.
![Page 29: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/29.jpg)
![Page 30: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/30.jpg)
• Represents a major cultural change.
![Page 31: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/31.jpg)
• Represents a major cultural change
• From an ‘accountability’ model to one centred on the corporate goal.
![Page 32: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/32.jpg)
• Represents a major cultural change
• From an ‘accountability’ model to one centred on the corporate goal.
![Page 33: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/33.jpg)
How?
![Page 34: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/34.jpg)
A marketingapproach
![Page 35: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/35.jpg)
A marketing approach
• Product life-cycle
![Page 36: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/36.jpg)
A marketing approach
• Product life-cycle
![Page 37: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/37.jpg)
A marketing approach
• Product life-cycle
• Customer
![Page 38: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/38.jpg)
A marketing approach
• Product life-cycle
• Customer– Needs and wants
![Page 39: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/39.jpg)
A marketing approach
• Product life-cycle
• Customer– Needs and wants
![Page 40: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/40.jpg)
A marketing approach
• Product life-cycle
• Customer– Needs and wants
• The grudge purchase
![Page 41: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/41.jpg)
A marketing approach
• Product life-cycle
• Customer– Needs and wants
• The grudge purchase
![Page 42: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/42.jpg)
A marketing approach
• Product life-cycle
• Customer– Needs and wants
• The grudge purchase
![Page 43: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/43.jpg)
A marketing approach
• Product life-cycle
• Customer– Needs and wants
• The grudge purchase
![Page 44: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/44.jpg)
A marketing approach
• Product life-cycle
• Customer– Needs and wants
• The grudge purchase
• Brands and brand equity
![Page 45: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/45.jpg)
A marketing approach
• Product life-cycle
• Customer– Needs and wants
• The grudge purchase
• Brands and brand equity
![Page 46: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/46.jpg)
A marketing approach
• Product life-cycle
• Customer– Needs and wants
• The grudge purchase
• Brands and brand equity
• Competition
![Page 47: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/47.jpg)
A marketing approach
• Product life-cycle
• Customer– Needs and wants
• The grudge purchase
• Brands and brand equity
• Competition
![Page 48: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/48.jpg)
A marketing approach
• Product life-cycle
• Customer– Needs and wants
• The grudge purchase
• Brands and brand equity
• Competition
• 4 Ps – Profit Product Place Promotion
![Page 49: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/49.jpg)
Priorities?
• To continue development & refinement of our ‘system’
• To address changing thefocus of our evaluation culture
• To move from ‘good oil’ to ‘best value’ oil
![Page 50: Bronte Koop, Principal Consultant, Strategy and Evaluation](https://reader036.vdocument.in/reader036/viewer/2022062422/568140cc550346895dac9970/html5/thumbnails/50.jpg)
Disclaimer
The views and interpretationsexpressed in this presentation are those of the authors anddo not necessarily reflect those of the Department of Education and Children's Services or the Government of South Australia