assessment tools and counting...2 mother’s hen is big and brown. it eats maize every morning and...
TRANSCRIPT
Assessment Tools and Counting
2 March 2012
Dr. Benjamin Piper
PRIMR Kenya
2
Mother’s hen is big and brown. It eats maize every morning and evening. It lays eggs in the fence.
One day, mother’s hen disappeared for three weeks. It came back with five chicks.
John was very happy to see the chicks. Mother said, “Don’t play with the chicks.” John did not listen. He played with the chicks.
Mother’s hen attacked John. He cried and ran to the house. He did not play with the chicks again.
Reading Fluency
3 wpm 20 wpm 30 wpm
Presentation Overview
• Importance of
Assessment
• Types of assessments
• Competency Benchmarks
• Deciding on meaningful
benchmarks in Grade 2
and 6 (GPE/USAID)
• Decline in non-readers
(GPE)
• Counting improved
readers (USAID)
• Fun add-ons for
assessment 3
Reading is a Global Movement
• Egypt
• Ethiopia
• Kenya
• Uganda
• Malawi
• Zambia
• South Africa
• DRC
• Ghana
• Liberia
• Mali
4
3/22/2012 5
Importance of Assessment
6
• You implement what you evaluate
• Story of reading evaluation in Country X
• Wasting money, wasting time
• Improving the systems’ focus on learning
• USAID and GPE require standard reading indicators
• Encouraging assessment throughout the system
Types of Assessments
• ASER
• UWEZO
• EGRA
• National assessments
7
• Key Question: Can one use them to
measure these reading activities?
8
ASER
9
ASER Results
ASER and Validity
• Research compared ASER with DIBELS
• If we accept the other tests as appropriate criterion then
the ASER language (and math tests) measure up well.
• Although a brief test, the ASER provides a valid
snapshot of early reading ability (and basic math ability)
11
UWEZO Kenya
12
UWEZO Kenya Reading Levels
13
UWEZO Uganda
EGRA (Early Grade Reading Assessment)
• Many segments timed to 1 minute
– Meant to measure fluency
• Entire test in less than 15 minutes
• Applied by assessor, one-on-one orally
• Can be done on sample basis
• Very easy to score
• Based on current research of critical elements of
reading
EGRA tasks in Kenya
KENYA
Task English Kiswahili Gikuyu Dholuo
Letter Naming X
Phonemic Awareness X X X X
Letter Sounds X X
Syllable Naming X X
Word Naming X X X X
Unfamiliar Word Naming X X X X
Oral Reading Fluency X X X X
Reading Comprehension X X X X
Listening Comprehension X X
3/22/2012
15
Letter Naming / Letter Sounds
3/22/2012 16
Phonemic Awareness
3/22/2012 17
Syllable Identification Fluency
nũ ru yi ye ri ke gi hu wo gũ
tĩ ri rũ ndũ rĩ ci nda cũ mwĩ gĩ
yũ ge thi mbũ mũ ka mwa ha ro ga
na ku wĩ wa hũ mĩ thĩ tha the mo
ti bũ ga tũ kĩ ya nya twa ba ga
na kĩ na ci thu mbe rĩ te hi gwa
ka mi rĩ ma nyi hi kĩ kũ gu ma
be ta gĩ ndĩ he hĩ mwe ne ya kwa
ki re to me co rwo ri ko tu mũ
go ho yo we mũ wa rĩ ka na re
Familiar Word Fluency
3/22/2012 19
Non-words
3/22/2012 20
Oral Reading Fluency and Reading Comprehension
3/22/2012 21
2011 Oral Reading Passage: P4 and P5
Yesterday I woke up very late. It was raining heavily
outside. When I looked out of the window I saw grandmother’s cows running towards Mr. Mukasa’s banana garden. They started
eating the banana plants before anyone could stop them!
I ran out of the house and tried to chase them back into
the kraal. I was not successful because they were many and they
were big. One of the cows kicked out at me and I ran back to the
house. Now I have a pain in the leg.
I know that Mr. Mukasa will complain to the Chairman.
Poor grandmother! I am sure she will have to pay for the plants
spoilt by her cows. She also needs to take me to the health
centre for treatment. Those cows caused a lot of trouble.
3/22/2012 22
National Assessments
Ethiopia Kenya Uganda
Organization NAEA&E KNEC UNEB
Assessment NLA NASMLA NAPE
Grade Level 4 & 8 3 & 6 3
Frequency Every 3-4 years Every 2-3 years Yearly
Sample Regionally
representative
Provincially
representative
Regionally
representative
Oral or written Written Written Written, with oral
components
Equated across
forms
By experts Yes By experts
23
Essential Research Methodology
24
• Random sampling
• Representative of population
• Sample size sufficient to detect changes
• School based or Household based
• Analyzable in terms of USAID/GPE indicators
Types of Assessments
• ASER
• UWEZO
• EGRA
• National assessments
25
• Key Question: Can one use them to
measure these reading activities?
• With some adjustments, yes
Key Indicators
26
• Reading competency at the end of
Grade 2 and Grade 6
• Decline in the proportion of non-readers
• Measuring the number of children with improved reading
Local Setting of Competency Benchmarks
• Based on actual data
• Done locally, with experts
• For each language
• Need to set benchmarks before program
start-up
27
Possible methods to set benchmarks
METHOD 1: Estimate what the average oral
reading fluency score is for children who
comprehend at 80% and 100%.
METHOD 2: Trend of scores who comprehend
well
METHOD 3: Average scores of the high
performing low SES schools
METHOD 4: Average scores if the particular
language was the language of instruction 75%-
80% of the time
Initial Attempts at Benchmarking
29
28.8
11.8
7.1 7.8
13.8
2.5 1.3
2.9 3.5
0.2 0 0 0
5
10
15
20
25
30
35
English Kiswahili Gikuyu Dholuo
Pe
rce
nta
ge a
t B
en
ch
ma
rk
Oral Reading Fluency in Kenya Compared with Benchmarks
45 wpm (Kenya Standard 2)
60 wpm (International Grade 2)
90 wpm (International Grade 3)
Oral Reading Fluency Benchmarks
Country Language Proposed Benchmark
Percent of children at benchmark (2010)
Target by 2015
Ethiopia Amharic 60 wpm 4.3% 80%
Afan Oromo 70 wpm 4.2% 60%
Tigrigna 60 wpm 0.4% 45%
Sidaamu Afoo 75 wpm 0.1% 70%
Hararigna 60 wpm 0.6% 50%
Kenya Dholuo 55 wpm 2.4%
Gikuyu 60 wpm 2.0%
English 100 wpm 3.6%
Kiswahili 65 wpm 2.4%
Uganda Luganda 60 wpm 0.4% 70%
Lango 55 wpm 0.0% 50% 3/22/2012 30
Oral Reading Fluency and Comprehension (Afaan Oromo)
3/22/2012 31
R squared .80
05
01
00
150
0 50 100 150Afan Oromo Oral Reading Fluency
Visual Inspection - Luganda
3/22/2012 32
02
04
06
08
01
00
0 20 40 60 80Luganda Oral Reading Fluency
Second Languages are More Difficult
3/22/2012 33
02
04
06
0
0 20 40 60 80English Oral Reading Fluency
Second languages are more difficult
3/22/2012 34
89 95
103
125
169.6
100 104.5
112
131.1
147.3
80
90
100
110
120
130
140
150
160
170
10th 25th 50th 75th 90th
English Oral Reading Fluency Scores Associated with High
Levels of Comprehension
RC = 80%
RC = 100%
Good News: Language Skills transfer
R^2=.63
020
40
60
80
0 20 40 60 80Luganda Oral Reading Fluency (Words Per Minute)
Suggestion…
• Use mother tongue for teaching
• Measure in mother tongue
• Compare across languages if necessary
36
Oral Reading Fluency by Reading Comprehension
3/22/2012 37
43.2
53.9 55.9 56.3 57.9 58.6
51.2
68.1 70.4
60.7
67.4
60
8.3
29.5 27.6 26.3 28.1
20
0
10
20
30
40
50
60
70
80
Sidamigna Amharic Hararigna Somali Afan Oromo
Tigrigna
80%
100%
Average
Oral Reading Fluency by Comprehension
91.7
59.8
39.2
55.2
99.2
64.1
50.7
66.1
30.4
21.8 20 19.6
0
20
40
60
80
100
120
English Kiswahili Gikuyu Dholuo
80%
100%
Mean
Oral Reading Fluency Benchmarks
Country Language Proposed Benchmark
Percent of children at benchmark (2010)
Target by 2015
Ethiopia Amharic 60 wpm 4.3% 80%
Afan Oromo 70 wpm 4.2% 60%
Tigrigna 60 wpm 0.4% 45%
Sidaamu Afoo 75 wpm 0.1% 70%
Hararigna 60 wpm 0.6% 50%
Kenya Dholuo 55 wpm 2.4%
Gikuyu 60 wpm 2.0%
English 100 wpm 3.6%
Kiswahili 65 wpm 2.4%
Uganda Luganda 60 wpm 0.4% 70%
Lango 55 wpm 0.0% 50% 3/22/2012 39
Decline in Non-readers (GPE)
• After the baseline
• Measure the percentage of non-readers
• Indicator: percentage of non-reader declines by half
• Measure at midterm and final
40
Kenyan Non-readers
0% 20% 40% 60% 80% 100%
English
Kiswahili
Gikuyu
Dholuo
14
19
15
20
40
42
55
54
42
37
28
24
3
3
2
2
Very low
Low
Moderate
Fluent
3/22/2012 41
Non-readers are quite high
3/22/2012 42
Hypothetical Example
29.7 27.5
41.2
26.5 33.7
69.2
17.9 10.1
33.9
12.7 17
20.6 21.4 14.8
54
9 3.8
20
0
10
20
30
40
50
60
70
80
Baseline
Midterm
3/22/2012 43
Counting
Measuring the number of children with improved reading skills
Baseline Hypothetical
3/22/2012 45
0% 20% 40% 60% 80% 100%
Sidama (SNNP)
Oromiya
Benishangul-Gumuz
Tigray
Amhara
Somali
Harari
Addis Ababa
Zero Low Moderate Benchmark or Above
Endline Hypothetical
3/22/2012 46
0% 20% 40% 60% 80% 100%
Sidama (SNNP)
Somali
Oromiya
Amhara
Benishangul-Gumuz
Tigray
Harari
Addis Ababa
Zero Low Moderate Benchmark or Above
Baseline Hypothetical
3/22/2012 47
0% 20% 40% 60% 80% 100%
Sidama (SNNP)
Oromiya
Benishangul-Gumuz
Tigray
Amhara
Somali
Harari
Addis Ababa
Zero Low Moderate Benchmark or Above
Endline Hypothetical
3/22/2012 48
0% 20% 40% 60% 80% 100%
Sidama (SNNP)
Somali
Oromiya
Amhara
Benishangul-Gumuz
Tigray
Harari
Addis Ababa
Zero Low Moderate Benchmark or Above
Most Frequent Scores by Level
English Central Region
English Lango region
Luganda Central Region
Lango Lango region
BL MT BL MT BL MT BL MT
0 53.1% 23.4% 87.9% 64.3% 50.8% 25.8% 81.8% 51.0%
1-10 32.5% 25.8% 10.6% 21.8% 23.6% 19.4% 15.0% 24.9%
11-20 8.7% 23.1% 1.4% 8.8% 13.9% 13% 3.0% 16.8%
20-29 2.3% 14.8% 0.0% 3.8% 8.9% 16.7% 0.0% 5.0%
30 + 3.4% 12.8% 0.0% 1.2% 2.7% 24.9% 0.2% 2.0%
AVERAGE 5.2 wpm 16.3 wpm 0.6 wpm 4.6 wpm 7.1 wpm 18.1 wpm 1.2 wpm 6.2 wpm
3/22/2012 49
Create your OWN benchmarks
Second Grade: Three Assessment Periods Per Year
Beginning of Year Months 1 - 3
Middle of Year Months 4 - 6
End of Year Months 7 - 10
DIBELS Measure
Scores Status Scores Status Scores Status
ORF 0 - 25
26 - 43
44 and above
At Risk
Some Risk
Low Risk
0 - 51
52 - 67
68 and above
At Risk
Some Risk
Low Risk
0 - 69
70 - 89
90 and above
At Risk
Some Risk
Low Risk
Third Grade: Three Assessment Periods Per Year
Beginning of Year Months 1 - 3
Middle of Year Months 4 - 6
End of Year Months 7 - 10
DIBELS Measure
Scores Status Scores Status Scores Status
ORF 0 - 52
53 - 76
77 and above
At Risk
Some Risk
Low Risk
0 - 66
67 - 91
92 and above
At Risk
Some Risk
Low Risk
0 - 79
80 - 109
110 and
above
At Risk
Some Risk
Low Risk
51
52
Fluency Alone is NOT Enough
39.8
26.6
20
25.6
18.6 19.6
9.3
47.6
54.3
6
17.6
56
0
10
20
30
40
50
60
English Kiswahili Gikuyu English Kiswahili Dholuo
Rural Central Rural Luo-Nyanza
Oral Reading Fluency Reading Comprehension
Designing an Assessment Program
2012 2013 2014
Cohort Term 1 Term 2 Term 3 Term 1 Term 2 Term 3 Term 1
1
2
3
53
Research Design
Baseline Midterm Final
Formative Assessment
Major Assessments
Organizing for Implementation
• Sample-based
EGRA/EGMA assessment
• Careful sampling frame
• Randomly selected
districts or zones
• Randomly selected
schools (clustered)
• Randomly selected
children
• Repeated at midterm and final
• Undertaken WITH support 54
Design Assessment Tools
• For each language
• Equated forms developed
• Equating done using IRT
theory
• Never translate from
another language or country
• Use linguists and curriculum
experts
• Adapt to local languages
and curricula
• Piloted using IRT theory to
improve assessment
55
Implement Baseline
• Often in a three stage
randomly stratified design
• In randomly selected
schools
• With randomly selected
children
• Using streamlined data
entry methods
• Analyzed using replicable
methods
56
Implement Midterm
• Midterm included to help
with program corrections
• Need to have the midterm
close to the baseline to
catch problems quickly
• Do NOT use the same
assessment
• DO equate the forms
• Use a variety of methods
of analysis
57
Use Assessment to Correct Course
• Grade level differences?
• Gender differences?
• Regional differences?
• Investigate teacher
behavior and take-up
• What incentives can you
create that are low or
no-cost?
• Travel allowances
• Investigate related
factors, intervene where
possible 58
59
Final Assessment
• Continue randomly
selecting schools
• Maybe include
longitudinal sub-sample
• Measure effect size and
program gains
• Report on successes and
failures
• Make datasets publicly
available
60
What is needed for all of this?
• Decisions on
assessments to use
• Skilled language people
(curriculum and linguists)
• Assessment department
skills
• Approximately 3-5% of
intervention budget
• Commitment to fast
results
• Commitment to unbiased
analysis
61
“Extras” for Assessment
• e-EGRA tools
• Assessment in classrooms
• Assessment to lead to “systems” change
• Revitalizing the reporting to focus on learning
62
Require Meaningful Measures
3/22/2012 www.rti.org 64
0
10
20
30
40
50
60
70
80
English Kiswahili Gikuyu Dholuo
0 25<1 50<25 75<50 >75