introduction to evaluation odette parry & sally-ann baker...
TRANSCRIPT
![Page 2: Introduction to Evaluation Odette Parry & Sally-Ann Baker o.parry@glyndwr.ac.uko.parry@glyndwr.ac.uk s.a.baker@glyndwr.ac.uks.a.baker@glyndwr.ac.uk](https://reader035.vdocument.in/reader035/viewer/2022062517/56649eaa5503460f94bb00a0/html5/thumbnails/2.jpg)
Aim and objectives of our presentation
Define evaluation and examine how it differs from research
Briefly introduce different evaluation designs and data collection approaches-Strengths and limitations
Developing evaluation plans Governance and ethical issues
![Page 3: Introduction to Evaluation Odette Parry & Sally-Ann Baker o.parry@glyndwr.ac.uko.parry@glyndwr.ac.uk s.a.baker@glyndwr.ac.uks.a.baker@glyndwr.ac.uk](https://reader035.vdocument.in/reader035/viewer/2022062517/56649eaa5503460f94bb00a0/html5/thumbnails/3.jpg)
What is evaluation?
Evaluation is “a set of procedures to judge a services merit by providing a systematic assessment of it’s aims, objectives, activities, outputs, outcomes and costs.” (NHS Executive, 1997)
Evaluation is learning about ‘what works’ and lessons for future development Evaluation determines how a service is doing
![Page 4: Introduction to Evaluation Odette Parry & Sally-Ann Baker o.parry@glyndwr.ac.uko.parry@glyndwr.ac.uk s.a.baker@glyndwr.ac.uks.a.baker@glyndwr.ac.uk](https://reader035.vdocument.in/reader035/viewer/2022062517/56649eaa5503460f94bb00a0/html5/thumbnails/4.jpg)
So why evaluate? Provides evidence of:
- Is what we are doing working?
- What are the benefits and impacts?
- What was successful and what not? And why?
- Have/are objectives being achieved?
![Page 5: Introduction to Evaluation Odette Parry & Sally-Ann Baker o.parry@glyndwr.ac.uko.parry@glyndwr.ac.uk s.a.baker@glyndwr.ac.uks.a.baker@glyndwr.ac.uk](https://reader035.vdocument.in/reader035/viewer/2022062517/56649eaa5503460f94bb00a0/html5/thumbnails/5.jpg)
So why evaluate?
Provides evidence for:
- Stakeholders
- Further programme development
- Staff development
- Other organisations
- Funders
![Page 6: Introduction to Evaluation Odette Parry & Sally-Ann Baker o.parry@glyndwr.ac.uko.parry@glyndwr.ac.uk s.a.baker@glyndwr.ac.uks.a.baker@glyndwr.ac.uk](https://reader035.vdocument.in/reader035/viewer/2022062517/56649eaa5503460f94bb00a0/html5/thumbnails/6.jpg)
What do you want to find out?
What is happening and how often?
How is it happening and why is it happening as it is?
![Page 7: Introduction to Evaluation Odette Parry & Sally-Ann Baker o.parry@glyndwr.ac.uko.parry@glyndwr.ac.uk s.a.baker@glyndwr.ac.uks.a.baker@glyndwr.ac.uk](https://reader035.vdocument.in/reader035/viewer/2022062517/56649eaa5503460f94bb00a0/html5/thumbnails/7.jpg)
Approaches
The ‘What and How Much’ question in evaluation addresses measurable ‘OUTCOMES’ associated with Quantitative approaches (e.g. RCTs and Surveys)
The ‘How and Why’ question addresses PROCESS, associated with Qualitative approaches (e.g. semi-structured interviews, observation and focus groups)
![Page 8: Introduction to Evaluation Odette Parry & Sally-Ann Baker o.parry@glyndwr.ac.uko.parry@glyndwr.ac.uk s.a.baker@glyndwr.ac.uks.a.baker@glyndwr.ac.uk](https://reader035.vdocument.in/reader035/viewer/2022062517/56649eaa5503460f94bb00a0/html5/thumbnails/8.jpg)
Horses for Courses
Evaluation usually requires both Outcome & Process information
Evaluation may Formative &/or Summative
Small scale evaluations, while most often qualitative, do collect some quantifiable data
![Page 9: Introduction to Evaluation Odette Parry & Sally-Ann Baker o.parry@glyndwr.ac.uko.parry@glyndwr.ac.uk s.a.baker@glyndwr.ac.uks.a.baker@glyndwr.ac.uk](https://reader035.vdocument.in/reader035/viewer/2022062517/56649eaa5503460f94bb00a0/html5/thumbnails/9.jpg)
Plan your evaluation
• Evaluation is not a ‘bolt on’• Evaluation key to informing project
development and delivery• Plan early• Involve funders, stakeholders in the
planning process
![Page 10: Introduction to Evaluation Odette Parry & Sally-Ann Baker o.parry@glyndwr.ac.uko.parry@glyndwr.ac.uk s.a.baker@glyndwr.ac.uks.a.baker@glyndwr.ac.uk](https://reader035.vdocument.in/reader035/viewer/2022062517/56649eaa5503460f94bb00a0/html5/thumbnails/10.jpg)
Define your purpose
Clearly define the purpose of the evaluation
What are the main aims and objectives
This is a key step in the planning process and will guide how the findings will be used
![Page 11: Introduction to Evaluation Odette Parry & Sally-Ann Baker o.parry@glyndwr.ac.uko.parry@glyndwr.ac.uk s.a.baker@glyndwr.ac.uks.a.baker@glyndwr.ac.uk](https://reader035.vdocument.in/reader035/viewer/2022062517/56649eaa5503460f94bb00a0/html5/thumbnails/11.jpg)
Develop an evaluation plan • What is being evaluated• Purpose of the evaluation• What is known already• The questions the evaluation will address• Evaluation method to be used• Who & Where will the evaluation take
place• Timescales• The resources available
![Page 12: Introduction to Evaluation Odette Parry & Sally-Ann Baker o.parry@glyndwr.ac.uko.parry@glyndwr.ac.uk s.a.baker@glyndwr.ac.uks.a.baker@glyndwr.ac.uk](https://reader035.vdocument.in/reader035/viewer/2022062517/56649eaa5503460f94bb00a0/html5/thumbnails/12.jpg)
Be realistic about what can be achieved
• Often evaluation is a compromise between the ideal and the achievable- the wants of different groups- constraints of methodology- evaluator skills- resource limitations- time limitations- ethical and governance issues
![Page 13: Introduction to Evaluation Odette Parry & Sally-Ann Baker o.parry@glyndwr.ac.uko.parry@glyndwr.ac.uk s.a.baker@glyndwr.ac.uks.a.baker@glyndwr.ac.uk](https://reader035.vdocument.in/reader035/viewer/2022062517/56649eaa5503460f94bb00a0/html5/thumbnails/13.jpg)
Different ways to collect data
Decide which methods to use in order to get the information you need. You may use one or more of the following:- Existing data- Document analysis- Interviews- Self completed questionnaires- Observation- Focus Group
![Page 14: Introduction to Evaluation Odette Parry & Sally-Ann Baker o.parry@glyndwr.ac.uko.parry@glyndwr.ac.uk s.a.baker@glyndwr.ac.uks.a.baker@glyndwr.ac.uk](https://reader035.vdocument.in/reader035/viewer/2022062517/56649eaa5503460f94bb00a0/html5/thumbnails/14.jpg)
Using existing data Use data routinely collected to examine
process and outcomes
Can save time but need to ensure that data collected, in a form that can be analysed and consistent with evaluation plan Evaluation of All Wales Dietetics Scheme used Minimum
data sets. Developed in conjunction with WAG and project dieticians to ensure consistency with evaluation aims
Collected data on project activity e.g. courses run, start and completion dates, description of activities, participant details, course outcomes, partnerships built etc
![Page 15: Introduction to Evaluation Odette Parry & Sally-Ann Baker o.parry@glyndwr.ac.uko.parry@glyndwr.ac.uk s.a.baker@glyndwr.ac.uks.a.baker@glyndwr.ac.uk](https://reader035.vdocument.in/reader035/viewer/2022062517/56649eaa5503460f94bb00a0/html5/thumbnails/15.jpg)
Document Analysis
Policy documents, minutes, operational policies- Track project development, aims of
project Whilst can provide rich information May only provide part of picture May be open to subjective
interpretation
![Page 16: Introduction to Evaluation Odette Parry & Sally-Ann Baker o.parry@glyndwr.ac.uko.parry@glyndwr.ac.uk s.a.baker@glyndwr.ac.uks.a.baker@glyndwr.ac.uk](https://reader035.vdocument.in/reader035/viewer/2022062517/56649eaa5503460f94bb00a0/html5/thumbnails/16.jpg)
Observation
Observing participants in activities
Participant vs non participant Observer effects Not suitable for some settings Can be difficult to collect & record
data
![Page 17: Introduction to Evaluation Odette Parry & Sally-Ann Baker o.parry@glyndwr.ac.uko.parry@glyndwr.ac.uk s.a.baker@glyndwr.ac.uks.a.baker@glyndwr.ac.uk](https://reader035.vdocument.in/reader035/viewer/2022062517/56649eaa5503460f94bb00a0/html5/thumbnails/17.jpg)
Interviews
Enables in depth exploration of how people think & feel about certain topics, effectiveness of your intervention etc
Rich data, allows in-depth understanding Can explore more sensitive area Can tailor to the individual No group influence Resource intensive
![Page 18: Introduction to Evaluation Odette Parry & Sally-Ann Baker o.parry@glyndwr.ac.uko.parry@glyndwr.ac.uk s.a.baker@glyndwr.ac.uks.a.baker@glyndwr.ac.uk](https://reader035.vdocument.in/reader035/viewer/2022062517/56649eaa5503460f94bb00a0/html5/thumbnails/18.jpg)
Focus groups
Investigation of how groups perceive topics, view the effectiveness of your intervention
Clarify issues identified in surveys Provide solutions to problems Less resource intensive and 1 to 1
interviews Group management issues Group process issues
![Page 19: Introduction to Evaluation Odette Parry & Sally-Ann Baker o.parry@glyndwr.ac.uko.parry@glyndwr.ac.uk s.a.baker@glyndwr.ac.uks.a.baker@glyndwr.ac.uk](https://reader035.vdocument.in/reader035/viewer/2022062517/56649eaa5503460f94bb00a0/html5/thumbnails/19.jpg)
Questionnaires
Identification of patterns, trends Investigate needs, expectations,
perspectives, preferences, satisfaction, knowledge
Large samples, relatively lower cost Ease of analysis Low response rates Respondent bias Language and literacy issues
![Page 20: Introduction to Evaluation Odette Parry & Sally-Ann Baker o.parry@glyndwr.ac.uko.parry@glyndwr.ac.uk s.a.baker@glyndwr.ac.uks.a.baker@glyndwr.ac.uk](https://reader035.vdocument.in/reader035/viewer/2022062517/56649eaa5503460f94bb00a0/html5/thumbnails/20.jpg)
Analysing your data
Avoid bias, involve more than one person in the task
Address your key questions Combine data types & findings from
different sources Compare views of different groups Don’t anticipate results, look for
unexpected findings
![Page 21: Introduction to Evaluation Odette Parry & Sally-Ann Baker o.parry@glyndwr.ac.uko.parry@glyndwr.ac.uk s.a.baker@glyndwr.ac.uks.a.baker@glyndwr.ac.uk](https://reader035.vdocument.in/reader035/viewer/2022062517/56649eaa5503460f94bb00a0/html5/thumbnails/21.jpg)
Evidence of impactFocus on whether:
the purpose has been achieved
the needs of those who take part have been met
there are unintended outcomes arising from the intervention
the intervention has resulted in changes in behaviour
there are barriers to and facilitators of successful implementation
![Page 22: Introduction to Evaluation Odette Parry & Sally-Ann Baker o.parry@glyndwr.ac.uko.parry@glyndwr.ac.uk s.a.baker@glyndwr.ac.uks.a.baker@glyndwr.ac.uk](https://reader035.vdocument.in/reader035/viewer/2022062517/56649eaa5503460f94bb00a0/html5/thumbnails/22.jpg)
Using your findings
Inform strategy, policy development Inform budgets Inform funding proposals Inform improvement plans & and make
changes Identification of training needs Identification of areas for future
research/evaluation
![Page 23: Introduction to Evaluation Odette Parry & Sally-Ann Baker o.parry@glyndwr.ac.uko.parry@glyndwr.ac.uk s.a.baker@glyndwr.ac.uks.a.baker@glyndwr.ac.uk](https://reader035.vdocument.in/reader035/viewer/2022062517/56649eaa5503460f94bb00a0/html5/thumbnails/23.jpg)
Presenting evaluation findings
Present balanced view, don’t just report the positive
Consider the needs of your audience, Different versions of reports or types of
presentation might be needed
![Page 24: Introduction to Evaluation Odette Parry & Sally-Ann Baker o.parry@glyndwr.ac.uko.parry@glyndwr.ac.uk s.a.baker@glyndwr.ac.uks.a.baker@glyndwr.ac.uk](https://reader035.vdocument.in/reader035/viewer/2022062517/56649eaa5503460f94bb00a0/html5/thumbnails/24.jpg)
Reflect on the evaluation process Were aims and objectives met? If not why Were methods employed appropriate?
Tools, recruitment methods, analysis etc
Did you reach target groups? Were resources sufficient? What changes resulted ? Future impacts?
![Page 25: Introduction to Evaluation Odette Parry & Sally-Ann Baker o.parry@glyndwr.ac.uko.parry@glyndwr.ac.uk s.a.baker@glyndwr.ac.uks.a.baker@glyndwr.ac.uk](https://reader035.vdocument.in/reader035/viewer/2022062517/56649eaa5503460f94bb00a0/html5/thumbnails/25.jpg)
Governance Issues
Be careful about crossing over from evaluation into the realms of research – carries additional connotations for ethics approval etc
Take advice Take an ethical/professional approach
to the evaluation, consider the participant and yourself
![Page 26: Introduction to Evaluation Odette Parry & Sally-Ann Baker o.parry@glyndwr.ac.uko.parry@glyndwr.ac.uk s.a.baker@glyndwr.ac.uks.a.baker@glyndwr.ac.uk](https://reader035.vdocument.in/reader035/viewer/2022062517/56649eaa5503460f94bb00a0/html5/thumbnails/26.jpg)
Ethics and the rights of Participants Requirements of Data Protection Act (1998) Some evaluations may require ethical
approvalIssues to consider- Informed Consent- Deception- Debrief- Withdrawal- Confidentiality
![Page 27: Introduction to Evaluation Odette Parry & Sally-Ann Baker o.parry@glyndwr.ac.uko.parry@glyndwr.ac.uk s.a.baker@glyndwr.ac.uks.a.baker@glyndwr.ac.uk](https://reader035.vdocument.in/reader035/viewer/2022062517/56649eaa5503460f94bb00a0/html5/thumbnails/27.jpg)
Data Protection Act 1998 – Your responsibility
Only hold genuinely required information Certain types of sensitive information carry added
restrictions and may only be used with permission Information should only be collected for the purposes for
which it was initially required Data must only be processed in accordance with the
legislation People must not be harmed by how their information is
used People have a right to view information being held about
them Information must be accurate, kept up to date, kept
secure and deleted when obsolete
![Page 28: Introduction to Evaluation Odette Parry & Sally-Ann Baker o.parry@glyndwr.ac.uko.parry@glyndwr.ac.uk s.a.baker@glyndwr.ac.uks.a.baker@glyndwr.ac.uk](https://reader035.vdocument.in/reader035/viewer/2022062517/56649eaa5503460f94bb00a0/html5/thumbnails/28.jpg)
In summary
Evaluation an essential component Need to plan Variety of approaches Methods used dependent on questions
to be asked Consideration of Governance issues
![Page 29: Introduction to Evaluation Odette Parry & Sally-Ann Baker o.parry@glyndwr.ac.uko.parry@glyndwr.ac.uk s.a.baker@glyndwr.ac.uks.a.baker@glyndwr.ac.uk](https://reader035.vdocument.in/reader035/viewer/2022062517/56649eaa5503460f94bb00a0/html5/thumbnails/29.jpg)
Resources
Data Protection Act see the information commissioners pages http://www.ico.gov.uk/what_we_cover/data_protection.aspx
National Research Ethics Service if your evaluation takes place on NHS premises or with Staff, patients ethical approval might be needed check with your local R&D office and LREC administrator http://www.nres.npsa.nhs.uk/applications/apply/
UK Evaluation Society has a useful resource page http://www.evaluation.org.uk/resources/online-resources.aspx