session 12: egra data preparation and analysis...november 12, 2015 session 12: egra data preparation...

November 12, 2015

Session 12:

EGRA Data Preparation and Analysis

Outline of Presentation

This presentation will cover the following topics:

• EGRA Data Preparation

• EGRA Data Analysis

• Expectations for Reporting

3

Data Processing: [If done correctly]

• Technically, most difficult aspect of the study.

– Requires extensive understanding of hierarchical data structure

and complex sample statistics.

– Requires a lot of previous hands-on experience.

– Requires knowledge of how the specific sample was drawn and

how the data were collected.

– Requires only 1 person to do it. 1-2 people to check it.

– Takes a relatively short period of time to finish once data

collection is complete.

– But should not be rushed. [Should be checked over several

times by 1-2 people]

• Once finished and checked, the hard part is over.

– Several people can analyze the data and help write the report. 4

Data Processing: [If done incorrectly]

5

Activity #1

• What two components make up an EGRA data set?

• [Hint] Think Rows and Columns…

6

Data Composition: Sample + Questionnaires/Assessment

7

Questionnaires Variables Columns

Sam

ple

Ob

serv

ati

on

s

Ro

ws

school Id age female Item 1 … Score

school1 Student1 10 1 0 0 33.33

school1 Student2 9 0 1 1 25



EGRA Data Preparation – Observation [Row] Checklist

• Game of Clue: Who-when-where.

– Check all other assessments.

• Who: Conducted by a real assessor.

• When: Not on a weekend or holiday.

• Where: A school that was randomly sampled.

• How Many: Actual counts in the data match with the expected

number of students

• Review incomplete assessments

9

Day # date

Day of week School code School name

CompletedEGRA

counts

Enumerator

counts enumerators1 10/21/13 Mon PS0102-088 OLASITI PR. SCHOOL 11 4 AROLE(2)//JMTOW(3)//KMPAN(3)//SELIA(3)1 10/21/13 Mon PS0105-018 KIMOSONU PR. SCHOOL 10 4 CMATO(3)//FHENJ(3)//MASU(1)//RUSAM(3)1 10/21/13 Mon PS0205-038 WISDOM PR. SCHOOL 12 4 FNYAK(3)//HKILO(3)//HMDUN(3)//TKIRI(3)1 10/21/13 Mon PS0405-047 KASANGA PR. SCHOOL 12 4 IKIWA(3)//MMLAA(3)//MNOL(3)//MPAZI(3)1 10/21/13 Mon PS0406-105 MATIGANJOLA PR. SCHOOL 12 4 JURAS(3)//LSIMBE(3)//MKYEJ(3)//NKIBO(3)1 10/21/13 Mon PS0503-030 HAPPY PR. SCHOOL 11 4 GHAMI(2)//GJOSE(3)//JMANY(3)//SYMSE(3)

1 10/21/13 Mon PS0505-187JIPE MOYO ENGLISH

MEDIUM 12 4 HSELE(3)//LBIJO(3)//PANDE(3)//RBUSY(3)

Activity #2:

• Circle 4 things in the table below that should be

checked for data quality. What could have caused

these inconsistencies?

10

Day

# date

Day of

week School code School name

Completed

EGRA

counts

Assessor

counts Assessor User Names

1 10/19/13 Sat PS0102-088

OLASITI PR.

SCHOOL 12 4 AROLE(3)//JMTOW(3)//KMPAN(3)//SELIA(3)

2

10/21/13 Mon PS0105-018

KIMOSONU

PR. SCHOOL 10 4 CMATO(3)//FHENJ(3)//MASU(1)//RUSAM(3)

2

10/21/13 Mon PS0105-031

MUGABE PR.

SCHOOL 2 1 MASU(2)

2

10/21/13 Mon PS0205-038

WISDOM PR.

SCHOOL 9 3 FNYAK(3)//HKILO(3)//TKIRI(3))

3

10/22/13 Tue PS0205-038

WISDOM PR.

SCHOOL 3 1 HMDUN(3)

3 10/22/13 Tue Practice Practice 12 4 JURAS(3)//LSIMBE(3)//MKYEJ(3)//NKIBO(3)

3

10/22/13 Tue PS0505-187

JIPE MOYO

ENGLISH

MEDIUM 12 4 HSELE(3)//LBIJO(3)//PANDE(3)//RBUSY(3)

EGRA Data Preparation

12

Name of

subtask

variable

Label for subtask

variable

Name of

subtask

timed

variable Label for subtask timed variable

letterLetter Idenfication (Names) clpm Correct Letter Names per Minute

letter_soundLetter Identification (Sounds) clspm Correct Letter Sounds per Minute

fam_word Familiar Word Reading cwpm Correct Words per Minute

invent_word Nonword Reading cnonwpm Correct Nonwords per Minute

oral_read Oral Reading Fluency orf Oral Reading Fluency

read_comp Reading Comprehension

list_compListening Comprehension

syll_soundSyllable Identification (Sounds) csspm Correct Syllable Sounds per Minute

oral_vocab Oral Vocabulary

vocab Vocabulary

maze Maze

dict Dictation

EGRA subtask variable nomenclature andnames of the timed score variables

EGRA Data Preparation

13

Item and score variables

Suffix Variable suffix label Possible values

1-# Item #

0 "Incorrect" 1 "Correct" . <missing> "Not asked/didn't attempt"

_score Raw Score 0 - # Items in Subtask

_attempted Total items attempted 0 - # Items in Subtask

_score_pcnt Percent Correct 0-100

_score_zero Zero score indicator0 "Score>0" 1 "Score=0"

_attempted_pcnt Percent Correct of Attempted 0-100

Activity #3: Calculate Oral Reading Fluency

14

Reading Passage

[Oral Reading Fluency]

Reading

Comprehension

One day, Juma lost his hat.6

What did Juma lose?

[his hat]

He was not happy. It was very cold. He looked into

his desk and on his chair. 23

Where did Juma look for his hat?

[his desk, chair/seat, classroom, under the big

tree, playground]

The hat was not there. He ran to the playground.33

Where did Juma run?

[the playground]

He looked under the big tree. It was not there. He

told the teacher he had lost his hat. The teacher

pointed to Juma’s head.

58

Where was Juma’s hat?

[on Juma’s head]

Juma laughed.

60

Why did Juma laugh?

[one or more of because the hat was on Juma’s

head / he felt silly / embarrassed ]

___ Number Incorrect ____ Number Correct

___ Time Taken

*Grade2 Tanzania National Cross-Section Study

English EGRA Reading Passage

EGRA Data Preparation – Timed Subtasks

15

The intention of a timed subtask in an EGRA or EGMA

instrument is to calculate the items per minute rate.

Subtask_per_minute=𝑠𝑢𝑏𝑡𝑎𝑠𝑘_𝑠𝑐𝑜𝑟𝑒

𝑠𝑒𝑐𝑜𝑛𝑑𝑠 𝑡𝑎𝑘𝑒𝑛 𝑡𝑜 𝑐𝑜𝑚𝑝𝑙𝑒𝑡𝑒 𝑠𝑢𝑏𝑡𝑎𝑠𝑘× 60

𝑠𝑒𝑐𝑜𝑛𝑑𝑠

𝑚𝑖𝑛𝑢𝑡𝑒

EGRA Data Preparation – Timed Subtasks Score Checks

• Check and Edit Potential Extreme Values

16

0

100

200

300

400

500

600

700

800

0 5 10 15 20 25 30 35 40 45 50 55 60

Timed Oral Reading Fluency Score

Tim

edO

ral R

ead

ing

Sco

re [

wp

m]

Time Taken (seconds)






18

EGRA Data Analysis - Definitions

Descriptive Analysis (non-inferential statistics)

Simply describes specifics of the sample

Inferential Analysis

Incorporates sample weights and the cluster effect to project the

sample estimates to the population from which the sample was

drawn.

19

Sample

Sample

Clu

ster

Eff

ect

Descriptive Analysis - Recommendation

• Describe the sample according to the sub-group level to

be reported in the main results section.

• Commonly report the basic demographic information.

– Gender, grade, age, region ect.

20

Tanzania 2013 National G2 EGRA Sample Counts by School’s Performance Band*

PerformanceBand

CountsStudents % of Sample

High 465 20.5

Medium 840 37.1

Low 961 42.4Total 2266 100.0

*Performance band based on Standard 7 PSLE for 2012

Inferential Statistics - Recommendation

• Describe the Population according to the sub-group

level to be reported in the main results section.

– If possible include the estimated population counts for the

subgroups.

21


PerformanceBand

CountsStudents

% of Sample

Estimated Population Count

% of Population

High 465 20.5 14,762 1.4Medium 840 37.1 121,666 11.4

Low 961 42.4 931,773 87.2Total 2266 100.0 1,068,201 100.0


Inferential Statistics - Goal

Samples will always have some uncertainty in the estimates

when projecting them to the population.

• Goal:

– Obtain unbiased means/percentages estimates that truly reflect

(representative of) the population. Sample Weights

– Obtain an appropriate level of precision for these estimates

based on the subgroup level you would like to report Sample

Size, Cluster Effect, Sample Methodology

• An appropriate 95%Confidence band width

• Be able to detect statistical significant (and appropriate) difference

between sub-groups.

22

Activity #4 True or False

Samples are drawn to answer the main research questions.

Samples are typically proportionally representative of the population.

Sample weights make the sample representative of the population.

23

True

False

True

Sample Weights

• Samples are drawn the answer the main research questions

• Typically NOT proportionally representative of the

population.

• This causes: Over/Under sampling of sub-groups makes the

sample NOT representative of the population.

• Sample weights make the sample representative of the

population

• Unweighted analysis produces biased estimates that will not

be representative of the population. 24

Sample Weights Representative

• Example: Tanzania Grade 3 2013

• Level of Report: School Performance Band

25


PerformanceBand

CountsStudents

% of Sample

% of Population

High 465 20.5 1.4

Medium 840 37.1 11.4

Low 961 42.4 87.2Total 2266 100.0 100.0


Notice any over sampling?

Any under sampling?

How could this affect the

National estimates?

National Estimates. Kiswahili English

Sample [Not weighted]

Population [Weighted]

25.7

17.8

23.19.4

Estimates should be reflective of the population






26

SampleC

lust

er

Eff

ect

Reporting

Before writing the EGRA research report, consideration needs to be given to

who the audience is.

Primary Audience

• USAID

• Country Ministry of Education personnel

Secondary Audience

• Other stakeholders, e.g. District Education Officers

• Education researchers

Understanding the audience and their needs enables the writer to structure

and write the report appropriately.

27

Reporting

• The report must explicitly define the population of interest in the

study and clearly explain the sample methodology.

• The report must clearly state the objectives of the study and its

limitations.

• The main findings should be presented in clear, concise, and

non-technical language.

• The main report should present summary findings of inferential

data analysis including:

– Means and percentages

– Standard errors and/or 95% Confidence Intervals

– Distributions

– Formal statically tests when appropriate (i.e. comparing difference in

means between subgroups)

28

Tanzania 2013 “Snap Shot” Survey

• Population

– All P2 pupils attending public schools on mainland Tanzania

(estimated ~1.1 million pupils) during the 2012-2013 school

year.

• Objective

– Obtain a national estimate for the P2 pupil reading ability and to

see if P2 pupils attending high performing schools read

significantly better than those attending medium, and low

performing schools.

Notes:

1. National estimate did not include Zanzibar and Pemba.

2. Performance Band: High, Medium, low are based on the

Standard 7 Primary School Leaving Exam (PSLE) in 2012 29

Example Of Report Table

– Means and percentages

– Standard errors or 95% Confidence Intervals

– Distributions

– Formal statically tests when appropriate (i.e. comparing

difference in means between subgroups)

30

Tanzania 2013 National G2 Kiswahili EGRA Mean Kiswahili Results by School Performance Band

PerformanceBand Mean

95% Confidence Interval

Difference in Mean

High* 40.8 (36.6, 44.9) 12.2

Medium^ 28.6 (24.6, 32.7) -Low* 16.1 (12.8, 19.4) 12.5

Overall 17.0 (14.5, 21.3) n/a`Performance band based on Standard 7 PSLE for 2012

^Reference subgroup from which other groups were compared*P-value < 0.05 in t-test difference in means

Should these estimates

be reflective of the sample

or the population?

Examples of Graph Distributions:

Probability Density & Cumulative

31

0

10

20

30

40

50

60

70

80

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100

Tanzania 2013 Grade 2 National Kiswahili Oral Reading Fluency

Ora

l Rea

din

g Fl

uen

cy

[Wo

rds/

min

]

Percentile of Grade 2 Students

27.7

13.0

19.4

15.714.3

5.72.9

0.9 0.60

5

10

15

20

25

30

0 1-10 11-20 21-30 31-40 41-50 51-60 61-70 71+


Oral Reading Fluency Score [Words/min]

% o

fG

rade

2 S

tude

nts

Probability Density by Performance Band

32

05

101520253035

0 1-10 11-20 21-30 31-40 41-50 51-60 61-70 71+

High Band

% o

f G

rade

2 S

tude

nts

Oral Reading Fluency Score [words/min]

-5

5

15

25

35

0 1-10 11-20 21-30 31-40 41-50 51-60 61-70 71+

Medium Band

% o

f G

rad

e2

Stu

de

nts


0

5

10

15

20

25

30

35

0 1-10 11-20 21-30 31-40 41-50 51-60 61-70 71+

Low Band%

of G

rade

2 S

tude

nts


Cumulative Distribution by Performance Band

33

0

10

20

30

40

50

60

70

80

90

0 10 20 30 40 50 60 70 80 90 100

Tanzania 2013 Grade 2 National Kiswahili Oral Reading Fluency by Performance band

High* Medium^ Low*

Ora

l R

eadin

g F

luency [

Word

s/m

in]

Percentile of Students

Reporting – Data Visualization

Data visualization should be used to facilitate understanding of the findings by

general audiences. Visualizations should be ‘standalone’, such that the visual

is interpretable without the audience needing to read extra text.

34

0

10

20

30

40

50

60

70

80

90

0 10 20 30 40 50 60 70 80 90 100

Tanzania 2013 Grade 2 National Kiswahili Oral Reading Fluency by Performance band

High* Medium^ Low*

Ora

l R

eadin

g F

luency [

Word

s/m

in]

Percentile of Students

Performance band based on 2012 Standard 7 PSLE examination

^Reference subgroup from which other groups were compared.

* P-values <0.05 t-test difference in means

PerformanceBand n

Wt%

MeanORF 95%CI

High* 465 1.4 40.8 (36.6, 44.9)

Medium^ 840 11.4 28.6 (24.6, 32.7)

Low* 961 87.2 16.1 (12.8, 19.4)

Over all 2266 100 17.9 (14.5, 21.3)

Reporting – Regression

• Linear Regression: – Continuous variable such as the Oral Reading Fluency score.

– Allows reports to conclude:

35

Tanzania 2013 National G2 Kiswahili EGRAMean Kiswahili Oral Reading Fluency Results by School Performance Band

PerformanceBand Mean

95% Confidence Interval

Difference in Mean

High* 40.8 (36.6, 44.9) 24.7

Medium* 28.6 (24.6, 32.7) 12.5

Low^ 16.1 (12.8, 19.4) -

`Performance band based on Standard 7 PSLE for 2012^Reference subgroup from which other groups were compared

*P-value < 0.05 in T-test difference in means

“Students attending medium performing schools read on average 12.5 words/min faster

than students attending low performing schools. Students attending high performing

schools read on average 24.7 words/min faster than students attending low performing

schools”.

Reporting

• The results should be reported by common

demographic variables and other variables of interest

as appropriate to the research questions.

• Results should be reported even if they are not

statistically significant.

• Difference between Actual Significant and Statistical

Significant

36

Activity #6: Circle the “Difference in Mean” that show actual/real

significant difference.

Tanzania 2013 National G2 Kiswahili EGRA

Mean Kiswahili Oral Reading Fluency Results by various demographics

Characteristic Sub-Group Mean

Difference in

Mean

95% Confidence

Interval

Performance

Band

High* 40.8 24.7 (36.6, 44.9)

Medium* 28.6 12.5 (24.6, 32.7)

Low^ 16.1 - (12.8, 19.4)

School Type’’Public^ 17.8 - (16.8, 18.5)

Private* 18.5 1.2 (17.8, 20,2)

GenderMale^ 16.2 - (12.8,19.6)

Female* 19.6 3.4 (15.8, 23,2)

PreschoolAttended 19.4 - (16.0, 22.6)

Did not

attend* 13.9 5.5 (10.6, 17.2)`Performance band based on Standard 7 PSLE for 2012

^Reference subgroup from which other groups were compared


`` Estimates were fabricated

37

Activity #6: Circle the “Difference in Mean” that show actual/real

significant difference.

National G2 Kiswahili EGRA

Mean Kiswahili Oral Reading Fluency Results

by years

Year`` Mean

Difference

in Mean

95% Confidence

Interval

2013^ 17.0 - (14.5, 21.3)

2015 15.7 -1.3 (13.6, 18.2)

2017* 23.5 6.5 (18.3, 25.8^Reference subgroup from which other groups were

compared


`` Estimates were fabricated

38

Examples of Non-Normal Distributions: Ghana 2015

40

0

10

20

30

40

50

60

70

80

90

0 1-10 11-20 21-30 31-40 41-50 51-60 61-70 71+

Ghana 2013 Grade2 Mother-TongueNzema Oral Reading Fluency

% o

fG

rade

2 S

tude

nts


Mean 3.0

Median 0

Mode 0

Logistic Regression

43

27.7

13.0

19.4

15.714.3

5.7

2.90.9 0.6

0

5

10

15

20

25

30

0 1-10 11-20 21-30 31-40 41-50 51-60 61-70 71+



% o

fG

rade

2 S

tude

nts

Mean 17.9

Median 16

Mode 0

• Use the data, find an appropriate Cut-Point

– Dichotomous “No/Yes”, (0,1) variables

– Report percentage by subgroup

Top ORF Reader

[Cut at 40 words/min]

0 1

Reporting – Logistic Regression

44

• Logistic Regression: – Report the percentages [rather than means]

– Report Odds-Ratios [rather than difference in means]

Tanzania 2013 National G2 Kiswahili EGRA Compare Top Kiswahili Readers' with Non-Top Readers by Performance Band’’

Performance Band

Not Top Orf Sampled

Top Orf Sampled

% TopReaders

Odds Ratio

High* 239 226 49.7 11.4Medium^ 670 170 21.1 3.1

Low* 866 95 8.0 -Overall 1775 491 10.0

`Top Kiswahili Readers defined as grade 2 students reading 40+ words/min‘’Performance band based on Standard 7 PSLE for 2012^Reference Subgroup from which other groups were compared*P-value < 0.05

“Students attending high performing schools are 11.4 times more likely to be top readers than students

attending middle performing schools. And students attending middle performance schools are 3.1 times

more likely to be top readers than students attending low performing schools”.

Reporting

• The following must accompany all reported inferential

estimates (including but not limited to means, median,

mode and proportions):

– Precision – either as 95% confidence interval for estimates, or

a t-score and p-value for comparisons in addition to standard

errors.

– Sample size

45

Note about Baseline Scores [Impact Evaluation Study]

46

Baseline

Mean T-Test P-Value

Effect

Size

Control 30.91.1 0.313 0.10

Treatment 32.3

Check for Control/Treatment Balance.

The difference is measured in standard deviations (called the ‘effect size’)

and should ideally be less than 0.25.

Baseline

Mean T-Test P-Value

Effect

Size

Control 30.95.3 <0.001 0.42

Treatment 22.3

Reporting: Not on Impact Evaluation Study

• Whenever results of comparisons of data across groups

at different times (baseline and end-line), effect size of

the difference needs to be reported.

48

Table 1: Example of Difference-in-Difference AnalysisBaseline Endline

Treatment

Mean

fluency

(wpm)

Standard

error

Number

of

sampled

students

t-statp-

value

Mean

fluency

(wpm)

Standard

error

Number

of

sampled

students

t-statp-

value

Differ-

ence–in–

difference

p-

value

(DiD)

Effect

size

Control 4.5 0.6 656 – – 9.5 1.6 475 – – – – –

Intervention 5.2 1.2 349 0.510 0.611 11.7 1.1 4801.18

90.236 1.5 0.490 0.12

Difference-in-Difference: (Mean endline treatment – mean baseline treatment) – (mean endline control – mean baseline control)

Effect Size (Cohen’s d): Difference-in-difference / pooled standard deviation

Reporting - Annexes

Researchers should also include details of the

methodology and results of the analysis is the

annexes, which can be quite lengthy when written

in a technical language. The following should be

included in the annexes:

49

Reporting - Annexes

50

1. Details of the methodology, methods and data

collection:

a. Study objectives

b. Design

c. Data collection methods and process

d. Data collection instruments

e. Method and results of equating if different tools were used

at different study points

f. Sampling parameters and attrition (for longitudinal studies)

g. Details on weighting

h. Limitations

i. Results of test reliability analysis (Cronbach’s alpha; item-

total correlations)

j. Intra-class correlation coefficient (ICC)

Reporting - Annexes

51

2. Details of analyses that were not included in the main

report:

a. Sample description

b. Details of descriptive analyses

c. Details of bivariate/multivariate analyses

Take-Home Suggestions

• EGRA Data Preparation– Have experienced statisticians do it right so that the data can be

analyze to infer about the population

• EGRA Data Analysis– Inferential analyze to project the estimates to the population.

• Expectations for Reporting– Report findings that answer the research questions, not just

statistically significant findings.

52

Sample

Clu

ster

Eff

ect

More Information

Chris Cummiskey

RTI International

[email protected]

Sample

Clu

ster

Eff

ect

mailto:[email protected]

session 12: egra data preparation and analysis...november 12, 2015 session 12: egra data preparation...

Documents