cecv intervention framework module 6 evaluation. purpose of this module as a result of participating...
TRANSCRIPT
![Page 1: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/1.jpg)
CECV Intervention Framework
Module 6 Evaluation
![Page 2: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/2.jpg)
Purpose of this Module
As a result of participating in this module, you will:• Evaluate the effectiveness of the Intervention Framework in guiding
your school through the process of:
Identifying students with additional learning needs; Assessing students with additional learning needs; Analysing & interpreting the data collected; Designing & carrying out the teaching; and Evaluating & monitoring the student’s progress and the
effectiveness of the teaching.
2
![Page 3: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/3.jpg)
Foundations of The Framework
3
![Page 4: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/4.jpg)
Core Principles
1. All students can succeed
2. Effective schools promote a culture of learning
3. Effective teachers are critical to student learning success
4. Teaching and learning is inclusive of all
5. Inclusive schools actively engage and work in partnership with the wider community
6. Fairness is not sameness
7. Effective teaching practices are evidence-based
4
![Page 5: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/5.jpg)
“…research seeks to prove, evaluation seeks to improve…”M Q Patton (2003)
5
![Page 6: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/6.jpg)
Focus Questions
• How do you define evaluation?
1. Why do you evaluate?
2. What do you evaluate?
3. For whom do you evaluate?
4. How do you evaluate?
Educational Evaluation Around the World, Danish Evaluation Institute, 2003
6
![Page 7: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/7.jpg)
How do you define
Evaluation?
7
![Page 8: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/8.jpg)
In implementing any change it is
necessary to evaluate the effect.
• In considering implementation of the Intervention Framework it is necessary to evaluate the effect on individual student outcomes and more broadly on teacher practice, teacher knowledge, school policies and processes.
8
Defining Evaluation
![Page 9: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/9.jpg)
Defining Evaluation
Evaluation involves assessing the strengths and weaknesses of programs, policies,
personnel, products, and organizations to improve their effectiveness.
The American Evaluation Association
9
![Page 10: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/10.jpg)
Defining Evaluation
Evaluation is the systematic collection and
analysis of data needed to make decisions,
a process in which most well-run programs
engage from the outset.
The American Evaluation Association
10
![Page 11: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/11.jpg)
Defining Evaluation
Evaluation is about finding answers to questions such as,
“are we doing the right thing” and “are we doing things right?”
The American Evaluation Association
11
![Page 12: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/12.jpg)
Defining Evaluation
Rossi and Freeman (1993) define evaluation as "the systematic application of social research procedures for assessing the conceptualization, design, implementation, and utility of ...
programs." 12
![Page 13: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/13.jpg)
Defining Evaluation• appraise• assess• critique• judge• justify• predict• prioritise• choose
Source: Anderson & Krathwohl, 2001
• monitor• select• rate• rank• prove• decide• conclude• argue
as cited in Biggs & Tang, 2007.
13
![Page 14: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/14.jpg)
Steps in Evaluation
Step 1: Define what you hope to achieveStep 2: Collect data (pre & post)Step 3: Analyse the dataStep 4: Formulate conclusionsStep 5: Modify the program
14
![Page 15: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/15.jpg)
Types of Evaluation
Process Evaluation Process Evaluations describe and assess the actual program materials and activities.
Outcome EvaluationOutcome Evaluations study the immediate or direct effects of the program on participants.
Impact Evaluation Impact Evaluations look beyond the immediate results of policies, instruction, or services to identify longer-term as well as unintended program effects.
15
![Page 16: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/16.jpg)
Process Evaluation to Inform School Improvement
Phases of the process of improvement0 Preparation/ 1. Identification1 Diagnostic phase / 2. Assessment2 Strategic planning phase / 3. Analysis and
Interpretation3 Developmental phase / 4. Teaching and
Learning4 Evaluation phase / 5. Evaluation
R Bollen 199716
![Page 17: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/17.jpg)
Outcome Evaluation
The ultimate goal of the Intervention Framework process is to improve student outcomes.
How do you know whether it did?• One commonly used way to find out whether the process (i.e. the
T&L cycle) improved student outcomes is to ask whether the process caused the expected outcome.
• If the process caused the outcome, then one could argue that the process improved student outcomes.
• On the other hand, if the process did not cause the outcome, then one would argue that, since the process did not cause the outcome, then the process did not improve student outcomes.
17
![Page 18: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/18.jpg)
Outcome Evaluation How to figure this out
Determining whether a process caused the outcome is one of the most difficult problems in evaluation, and not everyone agrees on how to do it. The approach you take depends on how the evaluation will be used, who it is for, what the evaluation users will accept as credible evidence of causality, what resources are available for the evaluation, and how important it is to establish causality with considerable confidence. Michael Quinn Patton
One way could be to evaluate the teaching programs implemented.
18
![Page 19: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/19.jpg)
Impact Evaluation
Impact Evaluations look beyond the immediate results of policies, instruction, or services to identify longer-term as well as unintended program effects.
19
![Page 20: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/20.jpg)
1. Why Evaluate?
20
![Page 21: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/21.jpg)
Why Evaluate?
It is important to evaluate programs/the teaching for many reasons:– to ensure that the program is not creating any
unintended harm;– to determine if the program is making a positive
contribution (improved student outcomes); and– to improve and learn (i.e. to learn what were the
positive elements, how it can be replicated, how challenges can be overcome in the future and how to make the process sustainable).
21
![Page 22: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/22.jpg)
Why Evaluate?
The four main reasons evaluation is conducted:accountability; learning; program management and development;ethical obligation.
Green and South, 2006.
22
![Page 23: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/23.jpg)
2. What do you Evaluate?
23
![Page 24: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/24.jpg)
3. For whom do you Evaluate?
24
![Page 25: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/25.jpg)
4. How do you Evaluate?
25
![Page 26: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/26.jpg)
What & How?
• How does your school evaluate its current programs?
• How would you evaluate whether the child/children progressed as a result of participation in this intervention process.
26
![Page 27: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/27.jpg)
• Is the student progressing satisfactorily against the set goals?
• How will you monitor and interpret the student’s progress against the set goals?
• How will you evaluate the effectiveness of the program/approach?
27
What & How?
![Page 28: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/28.jpg)
Making the results useful (student outcomes)
• How will you use the results to inform future program development for students?
• How will the results be reported so that they can be used by the school to make improvements?
28
What & How?
![Page 29: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/29.jpg)
“…evaluation seeks to improve…”
29
![Page 30: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/30.jpg)
Next Session - 2
30
![Page 31: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/31.jpg)
EvaluationEffect Sizes
31
![Page 32: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/32.jpg)
“evaluation seeks
to improve”
32
![Page 33: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/33.jpg)
Effect Sizes
1. What is an effect size?
2. Why use effect sizes?
3. How can schools use effect sizes to evaluate the effectiveness the intervention?
33
![Page 34: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/34.jpg)
Effect Sizes (d)
1a. What is an effect size?An effect size provides a common expression of the magnitude of study outcomes, across variables, such as improving reading levels in accuracy and comprehension.
An effect size of 1.0 indicates an increase of one standard deviation (1SD) on the outcome. One SD increase is typically associated with advancing students’ reading levels by two to three years, improving the rate of learning by 50%.
34
![Page 35: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/35.jpg)
Effect Sizes (d)
1b. What is a reasonable effect size?Cohen (1988) suggests that:d = 0.2 is small, d = 0.5 is medium, d = 0.8 is large
Whereas the results from Hattie’s meta-analysescould suggest when judging educationaloutcomes:d = 0.2 is small, d = 0.4 is medium, d = 0.6 is large
Reference: Cohen, J. (1988). Statistical power analysis for the behavioural sciences (2nd ed.). Hillsdale, NJ: L. Erlbaum Assoc.
35
![Page 36: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/36.jpg)
John Hattie - Visible Learning
What is John Hattie on about, in a nutshell?
15 years of research
800+ meta-analyses
50,000 studies
200+ million students
Outcome:
What are the major influences on student learning?
36
![Page 37: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/37.jpg)
Hattie’s Effect Sizes (d)
37
![Page 38: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/38.jpg)
Effect Sizes (d)
The Formula
Effect size = Average (post) - Average (Pre)
(d) Average Standard Deviation (the spread)
38
![Page 39: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/39.jpg)
Effect Sizes (d)
2. Why use effect sizes?• To compare progress over time on the
same test.
• To compare results measured on different tests.
• To compare different groups doing the same test.
39
![Page 40: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/40.jpg)
Effect Sizes (d)
3. How can schools use effect sizes?
Discussion in school groups
40
![Page 41: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/41.jpg)
EvaluationSelf Reflection
41
![Page 42: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/42.jpg)
What is
Self Reflection?
42
![Page 43: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/43.jpg)
Self Reflection
"Test all things; hold fast what is good"
I Thessalonians 5:21
43
![Page 44: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/44.jpg)
What is Self Reflection?
Self Reflection is simply a form of self evaluation undertaken by participants in social situations in order to improve the rationality and justice of their own practices, their understanding of these practices, and the situations in which the practices are carried out. Adapted from Carr and Kemmis, 1986
44
![Page 45: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/45.jpg)
Self Reflection in Schools
Self reflection is a process in which teachers examine their own educational practice systematically and carefully, using the techniques of action research.
45
![Page 46: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/46.jpg)
Self Reflection Leads to Improvement?
SELF REFLECTION
46
![Page 47: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/47.jpg)
Follow effective action with quiet reflection.
Out of the quiet reflection will come even more
effective action.Peter. F. Drucker
47
![Page 48: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/48.jpg)
Teachers account for about 30% of the variance in student achievement
“It is what teachers know, do, and care about, which is very powerful in this learning equation, not just what they know” (p. 2). They must put this knowledge into practice if they are to produce gains in student learning outcomes.
Hattie (2003)
48
![Page 49: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/49.jpg)
Where to from here?
49
![Page 50: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/50.jpg)
Where to from here?
Each school to reflect on:• Existing evaluation processes of:
– existing intervention programs currently in use in the school.
– teacher performance– Student performance
50
![Page 51: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/51.jpg)
In your Context
Has this process highlighted the need to review your school’s policies and/or processes?
51
![Page 52: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/52.jpg)
In your Context
• Has this process made a difference to your students’ performances?
Evaluation of change is fundamental to the process.
52
![Page 53: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/53.jpg)
“When ignorance is bliss, tis folly to be
wise”Thomas gray from ‘Ode on a Distant Prospect of Eton College’.
53
![Page 54: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/54.jpg)
“self reflection
seeks to
improve”
54
![Page 55: CECV Intervention Framework Module 6 Evaluation. Purpose of this Module As a result of participating in this module, you will: Evaluate the effectiveness](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dab5503460f94a9a944/html5/thumbnails/55.jpg)
References
• Educational Evaluation Around the World: An International Anthology
• http://english.eva.dk/publications/educational-evaluation-around-the-world-an-international-anthology/download
• An Education Primer: An overview of education evaluation http://www2.ed.gov/offices/OUS/PES/primer1.html
• Evaluation definition: http://www.evaluationwiki.org/index.php/Evaluation_Definition
• Evaluation Toolkit: http://mypeer.org.au/monitoring-evaluation
• Introduction to program Evaluation: http://www.cdc.gov/tb/programs/Evaluation/Guide/Webinar/Eval_101_1_AP22.ppt
• What is Program Evaluation? A beginner’s guide: http://gsociology.icaap.org/methods/evaluationbeginnersguide.pdf
• Patton, M Q 2002, Qualitative research & evaluation methods (3rd ed.), Thousand Oaks, CA: Sage Publications
55