barrow evidence computerized algebra instruction december 2007
DESCRIPTION
Barrow Evidence Computerized Algebra InstructionTRANSCRIPT
![Page 1: Barrow Evidence Computerized Algebra Instruction December 2007](https://reader036.vdocument.in/reader036/viewer/2022062410/5695d13e1a28ab9b0295b937/html5/thumbnails/1.jpg)
Technology’s Edge: The Educational Benefits of Computer-Aided Instruction
Lisa BarrowFederal Reserve Bank of Chicago
Lisa MarkmanPrinceton University
Cecilia RousePrinceton University and NBER
![Page 2: Barrow Evidence Computerized Algebra Instruction December 2007](https://reader036.vdocument.in/reader036/viewer/2022062410/5695d13e1a28ab9b0295b937/html5/thumbnails/2.jpg)
Student achievement is critical for individuals and society
• U.S. math skills have been increasing (NAEP), but proficiency levels remain low.
• Math skills may explain a large portion of wage inequality (Grogger, 1996; Murnane, Willet, & Levy, 1995)
![Page 3: Barrow Evidence Computerized Algebra Instruction December 2007](https://reader036.vdocument.in/reader036/viewer/2022062410/5695d13e1a28ab9b0295b937/html5/thumbnails/3.jpg)
School districts are turning to advances in computers to…
– Reduce administrative burden;
– Compensate for poor teacher content knowledge (especially in districts that report difficulty recruiting and retaining teachers, particularly in math and science);
– Allow more individualized student attention; students can progress at own pace.
![Page 4: Barrow Evidence Computerized Algebra Instruction December 2007](https://reader036.vdocument.in/reader036/viewer/2022062410/5695d13e1a28ab9b0295b937/html5/thumbnails/4.jpg)
Evidence on effectiveness of CAI is surprisingly weak
– Poorly defined computer use.
– Little use of randomized controlled study design.
![Page 5: Barrow Evidence Computerized Algebra Instruction December 2007](https://reader036.vdocument.in/reader036/viewer/2022062410/5695d13e1a28ab9b0295b937/html5/thumbnails/5.jpg)
In a 2001 review of the literature, Larry Cuban (2001, p. 179) concludes,
“When it comes to higher teacher and student productivity and a transformation of teaching and learning … there is little
ambiguity. Both must be tagged as failures. Computers have been oversold
and underused, at least for now.”
![Page 6: Barrow Evidence Computerized Algebra Instruction December 2007](https://reader036.vdocument.in/reader036/viewer/2022062410/5695d13e1a28ab9b0295b937/html5/thumbnails/6.jpg)
Computer-Aided Instruction (CAI) Treatment:
• Typically used in large urban districts;• Software and hardware package designed
to deliver one-on-one instruction;• Software described as “meeting National
Council of Teachers of Mathematics standards”;
• Software may be configured to state standards;
• Includes classroom management tools.
![Page 7: Barrow Evidence Computerized Algebra Instruction December 2007](https://reader036.vdocument.in/reader036/viewer/2022062410/5695d13e1a28ab9b0295b937/html5/thumbnails/7.jpg)
The Program
• 5 components per lesson:– Pretest– Review– Lesson– Cumulative review– Comprehensive test
• Required to achieve certain degree of mastery before advancing;
• Teachers can monitor student progress.
![Page 8: Barrow Evidence Computerized Algebra Instruction December 2007](https://reader036.vdocument.in/reader036/viewer/2022062410/5695d13e1a28ab9b0295b937/html5/thumbnails/8.jpg)
The design of this experiment…
• Randomly assign 8th and 9th grade classes in three large urban districts to be taught using computer-aided algebra and pre-algebra instruction.
• Assess the impact on statewide tests and tests designed to target algebra and pre-algebra skills.
• Note that the computer use was well-defined.
![Page 9: Barrow Evidence Computerized Algebra Instruction December 2007](https://reader036.vdocument.in/reader036/viewer/2022062410/5695d13e1a28ab9b0295b937/html5/thumbnails/9.jpg)
Research Design: Within-school random assignment to
CAI or traditional instruction at the classroom level in three districts
Research design addresses two forms of selection bias:
• Non-random assignment of students to CAI;• Non-random assignment of teachers to CAI.
![Page 10: Barrow Evidence Computerized Algebra Instruction December 2007](https://reader036.vdocument.in/reader036/viewer/2022062410/5695d13e1a28ab9b0295b937/html5/thumbnails/10.jpg)
Table 1: Characteristics of the Districts in Our Study
Top 100 Districts
3 Districts Combined District 1 District 2 District 3
# Students 112,807 62,507 ~68,000 ~22,000 ~97,000
% Female 48.8 49.4 49.7 48.8 49.3
% Native American 0.6 0.5 0.1 0.1 1.0
% Asian 7.1 3.1 1.9 0.8 4.4
% African American 28.1 69.5 93.6 40.3 59.4
% Hispanic 34.1 16.2 1.1 54.3 18.0
![Page 11: Barrow Evidence Computerized Algebra Instruction December 2007](https://reader036.vdocument.in/reader036/viewer/2022062410/5695d13e1a28ab9b0295b937/html5/thumbnails/11.jpg)
Implementation of Random Assignment
• Schools provided us with class schedules of pre-algebra and algebra classes;
• Given option of eliminating particular teachers or classes from the experiment;
• Randomization information provided to schools after students’ classes were scheduled.
![Page 12: Barrow Evidence Computerized Algebra Instruction December 2007](https://reader036.vdocument.in/reader036/viewer/2022062410/5695d13e1a28ab9b0295b937/html5/thumbnails/12.jpg)
Teacher Period 1 Period 2 Period 3 Period 4
A Alg 1 Alg 1A Alg 1
B Alg 1 Alg 1A Alg 1A
C Alg 1A Alg I
D Alg I Alg 1 Alg 1A
Randomized Evaluation of a Computerized Math Curriculum
Current School Schedule
![Page 13: Barrow Evidence Computerized Algebra Instruction December 2007](https://reader036.vdocument.in/reader036/viewer/2022062410/5695d13e1a28ab9b0295b937/html5/thumbnails/13.jpg)
Teacher Period 1 Period 2 Period 3 Period 4
A Alg 1 Alg 1A Alg 1
B Alg 1 Alg 1A Alg 1A
C Alg 1A Alg I
D Alg I Alg 1 Alg 1A
Teacher Period 1 Period 2 Period 3 Period 4
A Regular Regular LAB
B LAB LAB Regular
C Regular Regular
D Regular Regular LAB
Randomized Evaluation of a Computerized Math Curriculum
Current School Schedule
Below is a sample schedule that would be returned to each school after random assignment
![Page 14: Barrow Evidence Computerized Algebra Instruction December 2007](https://reader036.vdocument.in/reader036/viewer/2022062410/5695d13e1a28ab9b0295b937/html5/thumbnails/14.jpg)
Table 2a: Schools and Students in Study – District 1Relevant Schools
Schools in Study
Students in Study
Number of Students 29,603 8,148 973
% Grade 8 19.3 16.8 40.4
% Grade 9 18.0 18.3 47.2
% Grade 10 15.1 17.8 4.4
% Female 50.5 49.0 52.0
% African American 94.2 97.2 87.8
% Hispanic 1.0 0.8 0.8
% White 2.6 0.4 0.1
% Native American <0.1 <0.1 0.0
% Asian 2.2 1.6 1.8
% Missing demographic data
9.6
![Page 15: Barrow Evidence Computerized Algebra Instruction December 2007](https://reader036.vdocument.in/reader036/viewer/2022062410/5695d13e1a28ab9b0295b937/html5/thumbnails/15.jpg)
Table 2b: Schools and Students in Study – District 2Relevant Schools
Schools in Study
Students in Study
Number of Students 5,270 4,476 412
% Grade 8 2.3 0.0 0.0
% Grade 9 38.0 40.0 52.7
% Grade 10 22.0 23.2 31.8
% Female 48.4 48.2 46.7
% African American 43.6 42.0 47.1
% Hispanic 50.1 51.2 44.7
% White 5.5 5.9 6.6
% Native American 0.2 0.1 0.2
% Asian 0.7 0.8 0.5
% Missing demographic data
0.2
![Page 16: Barrow Evidence Computerized Algebra Instruction December 2007](https://reader036.vdocument.in/reader036/viewer/2022062410/5695d13e1a28ab9b0295b937/html5/thumbnails/16.jpg)
Table 2c: Schools and Students in Study – District 3Relevant Schools
Schools in Study
Students in Study
Number of Students 27,572 3,540 200
% Grade 8 1.4 0.0 3.5
% Grade 9 35.6 40.0 91.5
% Grade 10 23.3 25.1 3.0
% Female 49.9 47.6 47.7
% African American 61.1 92.5 94.5
% Hispanic 15.2 1.2 0.5
% White 18.3 4.0 1.5
% Native American 1.1 0.4 0.0
% Asian 4.5 1.9 3.0
% Missing demographic data
0.5
![Page 17: Barrow Evidence Computerized Algebra Instruction December 2007](https://reader036.vdocument.in/reader036/viewer/2022062410/5695d13e1a28ab9b0295b937/html5/thumbnails/17.jpg)
Numbers of Schools Classes, Teachers, and Randomization Pools
Analysis Sample
Combined District 1 District 2 District 3
Number of schools 17 10 4 3
Number of randomization pools
60 31 19 10
Number of classes 141 74 44 23
Number of teachers 57 36 14 7
Number of students 1,585 973 412 200
![Page 18: Barrow Evidence Computerized Algebra Instruction December 2007](https://reader036.vdocument.in/reader036/viewer/2022062410/5695d13e1a28ab9b0295b937/html5/thumbnails/18.jpg)
Outcomes
• Algebra and pre-algebra tests by Northwest Evaluation Association (NWEA) to be consistent with state and district standards;
• State-wide administered math tests;
• District benchmark tests in pre-algebra.
![Page 19: Barrow Evidence Computerized Algebra Instruction December 2007](https://reader036.vdocument.in/reader036/viewer/2022062410/5695d13e1a28ab9b0295b937/html5/thumbnails/19.jpg)
Table 3: Randomization of Treatment and Control Using Full Sample
Random Assignment
TraditionalInstruction CAI
p-value ofdiff.
Full Sample
Baseline algebra test score
24.7 24.7 0.494
Female 47.2 47.1 0.637
African American 80.0 83.2 0.561
Hispanic 15.9 13.5 0.195
Class Size 25.8 25.7 0.860
# of Observations 1,133 1,145
![Page 20: Barrow Evidence Computerized Algebra Instruction December 2007](https://reader036.vdocument.in/reader036/viewer/2022062410/5695d13e1a28ab9b0295b937/html5/thumbnails/20.jpg)
Table 3 (cont.): Randomization of Treatment and Control Using Analysis Sample
Random Assignment
TraditionalInstruction CAI
p-value ofdiff.
Analysis Sample
Baseline algebra test score
24.7 24.8 0.304
Female 51.1 48.9 0.148
African American 81.9 84.0 0.060
Hispanic 13.8 12.1 0.061
Class Size 25.8 26.2 0.549
# of Observations 785 800
![Page 21: Barrow Evidence Computerized Algebra Instruction December 2007](https://reader036.vdocument.in/reader036/viewer/2022062410/5695d13e1a28ab9b0295b937/html5/thumbnails/21.jpg)
Intent-to-Treat Estimates of the Effect of CAI on Algebra Achievement
(with and without Teacher Fixed Effects)
0.17
0.37
0.17
0.28
0
0.1
0.2
0.3
0.4
0.5
0.6
Without Teacher Fixed Effects With Teacher Fixed Effects
Est
imat
ed In
tent
-to-T
reat
Effe
ct S
ize
![Page 22: Barrow Evidence Computerized Algebra Instruction December 2007](https://reader036.vdocument.in/reader036/viewer/2022062410/5695d13e1a28ab9b0295b937/html5/thumbnails/22.jpg)
Advantages/Disadvantages of the Intent-to-Treat Effect
• Represents the gains a policy maker might reasonably expect to observe.
• Does not necessarily represent the effectiveness of the program.
![Page 23: Barrow Evidence Computerized Algebra Instruction December 2007](https://reader036.vdocument.in/reader036/viewer/2022062410/5695d13e1a28ab9b0295b937/html5/thumbnails/23.jpg)
Amount of Time in the Computer Lab by the Random Assignment of the Student’s Class
80.1
10.3 109.1
70.3
5.6
52.7
14.8
64.5
33
55.3
83.8
0
10
20
30
40
50
60
70
80
90
Number oflessons studentsare expected to
complete
% of stud.completing nolessons in CAI
% of stud.completing >10lessons in CAI
% of stud.completing >20lessons in CAI
Nr. of lessonscompleted in CAI
Nr. of CAI lessonscompleted as a %
of courseexpecations
Traditional Instruction CAI
![Page 24: Barrow Evidence Computerized Algebra Instruction December 2007](https://reader036.vdocument.in/reader036/viewer/2022062410/5695d13e1a28ab9b0295b937/html5/thumbnails/24.jpg)
Intent-to-Treat and Treatment on the Treated Estimates of the Effect of CAI
(with and without teacher fixed effects)
0.173
0.283
0.249
0.417
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Without Teacher Fixed Effects With Teacher Fixed Effects Inte
nt-to
-Tre
at a
nd T
reat
men
t-on-
the-
Trea
ted
Effe
ct S
izes
![Page 25: Barrow Evidence Computerized Algebra Instruction December 2007](https://reader036.vdocument.in/reader036/viewer/2022062410/5695d13e1a28ab9b0295b937/html5/thumbnails/25.jpg)
Intent-to-Treat Estimates in District 1 Using Different Tests
0.26
0.604
0.381
0.226
0
0.2
0.4
0.6
0.8
1
1.2
1.4
Algebra Scale Score 2nd Qtr Benchmark 3rd Qtr Benchmark State Math Test
Inte
nt-to
-Tre
at E
ffect
Siz
e
![Page 26: Barrow Evidence Computerized Algebra Instruction December 2007](https://reader036.vdocument.in/reader036/viewer/2022062410/5695d13e1a28ab9b0295b937/html5/thumbnails/26.jpg)
Intent-to-Treat Estimates in Districts 2 and 3 Using Different Tests
0.2
0.089
-0.124-0.062
-0.5
-0.4
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5
Algebra Scale Score State Math Test
Inte
nt-to
-Tre
at E
ffect
Siz
e
District 2 District 3
![Page 27: Barrow Evidence Computerized Algebra Instruction December 2007](https://reader036.vdocument.in/reader036/viewer/2022062410/5695d13e1a28ab9b0295b937/html5/thumbnails/27.jpg)
We might expect to see an advantage of CAI in…
• Classes where curriculum best suited to students;
• Larger classes;• Classes with more
disruptive students;• Classes with
heterogeneous students.
![Page 28: Barrow Evidence Computerized Algebra Instruction December 2007](https://reader036.vdocument.in/reader036/viewer/2022062410/5695d13e1a28ab9b0295b937/html5/thumbnails/28.jpg)
Differential Intent to Treat Effects of the Computerized Instruction on Pre-Algebra and
Algebra Achievement by Class Type
0.510.440.450.48
1.36
0.01 0.07 0.13
-0.31-0.23
-1.00
-0.50
0.00
0.50
1.00
1.50
2.00
2.50
3.00
All 3 districts Districts 1 & 2 District 1 District 2 District 3
Inte
nt-to
-Tre
at E
ffect
Siz
e
CAI Effect for Pre-Algebra CAI Effect for Algebra
![Page 29: Barrow Evidence Computerized Algebra Instruction December 2007](https://reader036.vdocument.in/reader036/viewer/2022062410/5695d13e1a28ab9b0295b937/html5/thumbnails/29.jpg)
Differential Intent to Treat Effects of CAI on Pre-Algebra and Algebra Achievement by
Baseline Test Score Quartile
0.2160.242
0.171 0.155
-0.1
0
0.1
0.2
0.3
0.4
0.5
Bottom baseline test scorequartile
2nd baseline test scorequartile
3rd baseline test scorequartile
Top baseline test scorequartile
![Page 30: Barrow Evidence Computerized Algebra Instruction December 2007](https://reader036.vdocument.in/reader036/viewer/2022062410/5695d13e1a28ab9b0295b937/html5/thumbnails/30.jpg)
Differential Intent to Treat Effects of CAI on Pre-Algebra and Algebra Achievement by
Individual Attendance Rates
0.439
-0.221 -0.051 -0.02
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
1.2
Bottom baselineattendance quartile
2nd baseline attendancequartile
3rd baseline attendancequartile
Top baseline attendancequartile
![Page 31: Barrow Evidence Computerized Algebra Instruction December 2007](https://reader036.vdocument.in/reader036/viewer/2022062410/5695d13e1a28ab9b0295b937/html5/thumbnails/31.jpg)
Differential Intent-to-Treat Effects by Class Characteristic: Attendance, Class Size, and Class
Baseline Test Score S.D.
0.06
0.350.21
0.01
-1.1
0.412
-1.2
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
ClassAvg.
Attend.Rate
Attnd.Rate OneSD below
Mean
Class size=25
Class size=15
Smallclasses
Largeclasses
Est
imat
ed In
tent
-to-T
reat
Effe
ct
![Page 32: Barrow Evidence Computerized Algebra Instruction December 2007](https://reader036.vdocument.in/reader036/viewer/2022062410/5695d13e1a28ab9b0295b937/html5/thumbnails/32.jpg)
Overall we find:
• On average, students in classes taught using CAI scored higher on algebra tests than students in traditionally-taught classes.
• The effects appear larger for students in larger classes (especially large, heterogenous classes), those with worse attendance rates, and those in classes with lower average attendance rates.
![Page 33: Barrow Evidence Computerized Algebra Instruction December 2007](https://reader036.vdocument.in/reader036/viewer/2022062410/5695d13e1a28ab9b0295b937/html5/thumbnails/33.jpg)
Cost-Benefit Analysis
Cost of CAI
# ClassesTotal
studentsClass size # periods
CAI labs needed
Cost/ student
School A 22 730 33.2 8 3.0 $218
School B 12 321 26.8 8 1.5 $245
District 1 74 1736 23.5 8 9.3 $279
Cost of reducing class size to 13
# ClassesTotal
studentsClass size # periods
# new teachers
Cost/ student
School A 22 730 33.2 6 5.7 $329
School B 12 321 26.8 6 2.1 $278
District 1 74 1736 23.5 6 9.9 $241