whole day first grade program: effectiveness, sustaining, & building a base for going to scale...
TRANSCRIPT
Whole Day First Grade Program: Effectiveness, Sustaining, &
Building a Base for Going to Scale
Jeanne Poduska, ScD
Center for Integrating Education and Prevention Research in Schools, American Institutes for Research
Presented at: New Directions for Mental Health and Drug Abuse Effectiveness
and Dissemination Trial Research and MethodologyOctober 28, 2005
Contributors and Support
Sheppard G. Kellam, M.D. 1
Jeanne Poduska, Sc.D. 1
C. Hendricks Brown, Ph.D. 2
Carla Ford, Ph.D. 1
Amy Windham, Ph.D. 1
Natalie L. Keegan 1
John Reid, Ph.D. 3
Bonnie Copeland, Ph.D. 4
Linda Chinnia, 4
National Institute on Drug Abuse,Division of Epidemiology, Services and Prevention Research,
Prevention Research Branch 1R01 DA015409 and 1R01 DA019984 (and support from NICHD)
1 American Institutes for Research, Center for Integrating Education and Prevention Research in Schools2 University of South Florida, Prevention Science Methodology Group
3 Oregon Social Learning Center4 Baltimore City Public School System
Efficacy
Effectiveness
Sustainability
Going to Scale
Sustaining Systemwide
Efficacy
Effectiveness
Sustainability
Going-to-Scale
Sustaining
Systemwide
Phases of Education and Prevention Trials
Developmental Epidemiology: A Strategy for Prevention Research
Community Epidemiology
Life Course Development
Preventive Interventions
Research roots of WD• First Generation of Baltimore trials tested separately
Mastery Learning (ML) aimed at reading, and Good Behavior Game (GBG) aimed at classroom management of student behavior in First Grade.
– Each had short-term impact on its target; GBG had long term impact on very high risk children.
• Second Generation tested combination of instructional improvement and GBG and showed impact on reading and behavior into middle school, still room for improvement.
– A family/classroom program also seemed promising.
• Oregon Prevention Research Center tested family/ classroom partnership (LIFT program) and found impact among higher risk children.
Add Outcome Slide
How do you Establish Community & Institutional Partnerships?
1. Analyze social/political structure
2. Learn the vision of problems and priorities
3. Identify mutual self-interest within and across leadership
4. Fit your interest under the vision of leadership
5. Request ad-hoc oversight committee of leaders
6. Work through trust
This process takes about a year
Functions of a Community & Institutional Board
• Establish vision and priorities
• Examine/critique/approve/support proposed programs
• Communicate constituents’ concerns, values, priorities, and acceptable language to program leaders
• Communicate program needs to constituent leaders
• Continually assess absent constituencies
• Work toward institutionalizing programs
Baltimore City Public School System Partners
Patricia Welch, Ph.D., Chair, BCPSS Board of School Commissioners
Bonnie Copeland, Ph.D., Chief Executive Officer
Linda Chinnia, Chief Academic Officer
Ben Feldman, Ph.D., Research, Evaluation & Accountability Officer
Sue Cutter, Area 1 Interim Academic Officer
James Smith, Area 2 Academic Officer
Gwen Cleage, Curriculum and Instruction Officer
La Verne Sykes, Director, Parent and Community Involvement
Deborah Wortham, Ph.D., Director of Professional Development
Delores Alexander, Coordinator of Reading
Marietta English, President, Baltimore Teachers Union
James Gittings, Public School Supervisors and Administrators Association
20 Schools, Principals, and Teachers Areas 1 & 2
Three Foci of Baltimore Whole Day First Grade Program (WD)
• Improve reading instruction in all academic subjects thru the day, based on prior Baltimore trials.
• Improve classroom behavior management, based on prior Baltimore trials.
• Improve parent/classroom partnership re reading and behavior, based on prior Baltimore and Eugene, Oregon trials.
Lessons Learned From First Two Generations of Trials in Baltimore
Without a system to mentor, model, and monitor teacher practices over time, practices are not sustained.
A multi-level institutional structure with mentoring and monitoring is required to implement interventions with fidelity.
Teachers need support from principals; principals from area leaders; area leaders from chief academic and executive officers.
Goals of WD Trial
• Can we find synergy from the combination?
• Can the practices be sustained over time?
• Can practices be scaled up by school system with same effect?
Baltimore Whole Day First Grade Program Analytic Model
ClassroomBehavior
Management Classrooms
DecreasedAggressive, Disruptive
Classrooms
Teachers’ EffectiveAcademic
Instruction
PoorReading Skills
ImprovedReading Skills
Decreased Later Conduct & Anti-
social PersonalityDisorders
Effective Family-Classroom
Partnerships
•••
Other mediating or moderating variables:
• Family structure and poverty• Deviant peersWhole DayWhole Day First Grade
Education and Prevention Program
Decreased Later DepressiveDisorders
Depressive Symptoms
DecreasedDepressive Symptoms
PoorAchievement
IncreasedAchievement
Decreased Later Substance Abuse
Individual Aggression
DecreasedIndividual Aggression
School Success & Decreased
Drop-Out
• School building• Community economic health, resources, drugs, and violence
Multi-Level Structure to Sustain Practices and Go to Scale BALTIMORE CITY PUBLIC SCHOOL SYSTEM
Board of School Commissioners
Chief Executive Officer
Chief Academic Officer
ProfessionalDevelopment Officer
Curriculum &Instruction Officer
Area AcademicOfficers
Lead Coaches
Principals Coaches
Teachers
Research and Evaluation Office
Core Team
BALTIMORE CITY PUBLIC SCHOOL SYSTEM
Board of School Commissioner
Chief Executive Officer
Chief Academic Officer
ProfessionalDevelopment Officer
Curriculum &Instruction Officer
Area AcademicOfficers
Lead Coaches
Principals Coaches
Teachers
Research and Evaluation Office
Schools Committee
BALTIMORE CITY PUBLIC SCHOOL SYSTEM
Board of School Commissioner
Chief Executive Officer
Chief Academic Officer
ProfessionalDevelopment Officer
Curriculum &Instruction Officer
Area AcademicOfficers
Lead Coaches
Principals Coaches
Teachers
Research and Evaluation Office
Within-school Random Assignment* of Teachers/Classrooms and Children in the
Whole Day First Grade Program
IN EACH SCHOOL:
WD
SC (a) SC (b)
Random
Random
Children randomly assigned to classrooms
Teachers randomly assigned to classrooms
*Brown & Liao. AJCP, 1999.
Effectiveness Design of the Whole Day First Grade Program:
Follow 2 Cohorts of 1st Graders to 3rd Grade2003-04 2004-05 2005-06
12TeachersWD
Cohort 1 Grade 1
Cohort 1 Grade 2
Cohort 2 Grade 1
Cohort 1 Grade 3
Cohort 2 Grade 2
12 TeachersSC
Cohort 1 Grade 1 Cohort 1 Grade 2
Cohort 2 Grade 1
Cohort 1 Grade 3
Cohort 2 Grade 2
Remaining Teachers
Sustainability Design of the Whole Day First Grade Program:
Follow Teachers over Cohorts of 1st Graders2003-04 2004-05 2005-06
12TeachersWD
Cohort 1 Grade 1
Cohort 2 Grade 1
Cohort 3 Grade 1
12 TeachersSC
Cohort 1 Grade 1
Cohort 2 Grade 1
Remaining Teachers
Scaling-Up Design of the Whole Day First Grade Program:
3 Cohorts of 1st Graders2003-04 2004-05 2005-06
12TeachersWD Cohort 1 Grade 1
Cohort 2 Grade 1Cohort 3 Grade 1
12 TeachersSCTrained in WD
Cohort 1 Grade 1Cohort 2 Grade 1
Cohort 3 Grade 1
Remaining TeachersTrained in WD
Cohort 3 Grade 1
Design of the Whole Day First Grade Program
2003-04 2004-05 2005-06
12TeachersWD
Effectiveness
Cohort 1 Grade 1
Effectiveness
Sustainability
Cohort 1 Grade 2
Cohort 2 Grade 1
Effectiveness Sustainability
Cohort 1 Grade 3
Cohort 2 Grade 2
Cohort 3 Grade 1
12 TeachersSC
Effectiveness
Cohort 1 Grade 1
Effectiveness
Sustainability
Cohort 1 Grade 2
Cohort 2 Grade 1
Effectiveness Scaling Up (to WD)
Cohort 1 Grade 3
Cohort 2 Grade 2
Cohort 3 Grade 1
Remaining Teachers
Scaling Up (to WD)
Cohort 3 Grade 1
Primary Measurement Objectives in the WD Trial
• Implementation
• Proximal effects of intervention on student behavior, achievement, and psychological well-being
• Longer term (distal) effects
• Teacher practices
• Hypothesized or known mediating and moderating variables
• Measures of the multi-level structure
Fidelity of Whole Day First Grade Program Implementation Checklist Excerpt
CLASSROOM BEHAVIOR MANAGEMENT INDICATORS QUALITY RATING
20. Instructs students on class rules by:
Announcing the game and reviewing rules.
Setting and adhering to time limits for the game.
Providing appropriate independent assignments during the game.
Y
Y
Y
Y
N
N
N
N
1 2 3 4 5 6
Classroom Behavior Management Global Rating:Global Rating is not necessarily the sum of individual quality ratings.
1 2 3 4 5 6
Measures of Effectiveness Social Adaptational Status
OUTCOME MODE MEASURE
ACADEMIC ACHIEVT Student interview Woodcock Johnson
Houston Word Reading Scales
Teacher Interview Teacher Observation of Classroom Adaptation (TOCA)
Record review School records, Standardized scores
BEHAVIOR
Time on-task/off-task Classroom Obs. Teacher Observation/Student Engagement (TOSE) - Foorman
Aggressive behavior Classroom Obs. Teacher Interview
TOSETOCA
Disruptive behavior Classroom Obs. Teacher Interview
TOSETOCA
Concentration prob’s Classroom Obs. Teacher Interview
TOSETOCA
Social Isolation Classroom Obs. Teacher Interview
TOSETOCA
Measures of Effectiveness Psychological Well-being
OUTCOME MODE MEASURE
Depression Student interview Direct Clinical ObservationBaltimore How I Feel
Anxiety Student interview Direct Clinical ObservationBaltimore How I Feel
Flat affect Student interview Direct Clinical Observation
Hyperkinesis Student interview Direct Clinical Observation
Bizarre behavior Student interview Direct Clinical Observation
Teacher Practices
Observations by Independent Observers
(3x per year)
• Teacher Practices
• Timed Observation of Instruction (Foorman)
• Global ratings
Measures of Multi-Level System to Support Sustaining
Practices and Scaling-Up
• Defined expectations for each level of the multi-level structure along four organizational variables: power/structure, tasks, resources, human factor
• Attendance logs and observers’ ratings of occurrence and quality for expectations
• 1 Core Team; 2 AAOs; 12 school teams
Chief Academic Officer
Core Team
Area Academic Officers &Lead Coaches
School Coaches / Principals
Monitoring and Mentoring Feedback Loop
MentoringMonitoring
Data from Monitoring
Effective Mentoring
Teacher Practices
CLASSROOM BEHAVIOR MANAGEMENT INDICATORS QUALITY RATING
22. Plays the Good Behavior Game by:
Dividing students into balanced teams.
Reviewing class rules and announcing the game.
Setting and adhering to time limits for the game.
Placing checks on the board when individual students break rules.
Providing rewards to winning teams.
Y
Y
Y
Y
Y
Y
N
N
N
N
N
N
1 2 3 4 5 6
Classroom Behavior Management Global Rating:Global Rating is not necessarily the sum of individual quality ratings.
1 2 3 4 5 6
Fidelity of Whole Day First Grade Program Implementation Checklist Excerpt
Patterns of Quality of GBG Implementation in
12 Intervention Schools: 2004-05
High4 - 6
▲▲▲▲▲▲
▲▲▲▲▲▲
▲▲▲▲▲▲
▲▲▲▲▲▲
Low1 - 3
11 18 25 36
Week of Implementation
Leve
l of
Impl
emen
tatio
n
Patterns of Quality of GBG Implementation in
12 Intervention Schools: 2004-05
High4 - 6 ● ● ● ● ● ● ● ● ●
Low1 - 3
● ● ● ● ● ● ●
11 18 25 36
Week of Implementation
Leve
l of
Impl
emen
tatio
n
Patterns of Quality of GBG Implementation in
12 Intervention Schools: 2004-05
High4 - 6
■ ■ ■
Low1 - 3 ■ ■ ■ ■ ■
11 18 25 36
Week of Implementation
Leve
l of
Impl
emen
tatio
n
Patterns of Quality of GBG Implementation in
12 Intervention Schools: 2004-05
High4 - 6
▲▲▲▲▲▲
▲▲▲▲▲▲
● ●■ ■
▲▲▲▲▲▲
● ● ●■
▲▲▲▲▲▲
● ● ● ●
Low1 - 3
● ● ● ●■ ■
● ● ●■
■ ■
11 18 25 36Week of Implementation
Leve
l of
Impl
emen
tatio
n
Professional Development Model as Moving to Scaling Up
• AIR (Carla Ford) transitioning from classroom/school based focus to multi-level professional development
• Carla monitoring coaches; coaches monitoring teachers
• 3 Area Lead Coaches being trained fall 2005; train teachers in spring 2006
• Developing monitoring system
Summary—Prevention and Service Researchers
Researcher/Community Institution Partnerships
Defining intervention and practices with precision
Defining and measuring multi-level structure for sustaining and scaling-up practices
Continual monitoring of practices deliberately linked to professional development
Dual design requires equal emphasis on measurement of effectiveness and going to scale
Summary—Statisticians and Methodologists
• Modeling mediation of teacher practices
• Relationship of observational data on teacher practices and checklist of teacher practices: thick to thin data
• Modeling multi-level structure data
• Help us prepare for missing data in follow-up!